For granularity, you can add a 2nd dimension to the category like so:
scarves:silk
scarves:cotton
t-shirts:cotton
t-shirt:linen
etc...
Now, when the user navigates your site, the category affinity array could look like this:
[scarves:cotton, t-shirts:cotton, t-shirts:linen, t-shirts:silk, bedsheets:cotton]
With this, you can see the visitor has an affinity for t-shirts, and also affinity for cotton products.