There should be an option to filter out all spider and bot traffic from SiteCatalyst reports, very similar to what was available from the HBX User Agent Filtering option.
I think this should include a warning system that identifies suspicious IP Addresses (i.e. sening an enormous amount of traffic per day) and allow you to block that IP address from now on...
This is a great idea. We currently do this via a VISTA rule but this is at an additional cost, I think this should come as standard within Site Catalyst.
We do see bots in our reporting that apparently execute the Java Script. Not many, but they are there. At this time, they do not seriously affect our reporting.
Have had this discussion/debate with omniture. They SHOULD be filtering out sipders without cost. One of the most important aspects of analytics is it's reliability, if bots/spiders are executing javascript or making image request (some implementation include this to capture non javascript users) then this places a big question mark over the numbers and requires further auditing. We have in the past spotted/identified bots/spiders executing javascript and/or making image requests, doesn't really matter what the % of bots/spiders that do this might be as one hit by any of these would skew up the data.
I have to agree that this is a big issue and an easy one to solve for Omniture. THey should simply maintain a list of known bot IPs and have a setting in the admin consule where we can click to have this IP filter applied to our data like a standard Vista Rule. And just like Vista rules; they would still get the server calls since it did hit their servers but we won't see the numbers in our data sets. That's a win / win for both parties. Clean data for us continued revenue for Omniture.
Bot/ spider filtering is critical for data accuracy, but we found out the hard way that this is not being done by Omniture. The cost to implement this via a Vista rule was also a surprise.
We requested that Omniture recreate the filter that was previously available through HBX, to no avail. In the end, we had to supply the bot/ spider list ourselves. They made a short list available, but it was incomplete and was not being updated.
Omniture should be using the IAB spider/ bot list (with regular updates) and implement this for all clients automatically.
Omniture can do this using a Vista rule, it is just a mnual task that needs to be done monthly. They can either use the IAB list of bots/spiders or in the UK ABCe publish a list of these. We are currently testing this facility which I believe should be offered to all customers as a matter of course - out of the box with no additional charge. Without credible data any insight is meaningless.
