Filter out Spider and Bot traffic from SiteCatalyst

Avatar

Avatar

vabeachkevin1

Avatar

vabeachkevin1

vabeachkevin1

11-02-2010

There should be an option to filter out all spider and bot traffic from SiteCatalyst reports, very similar to what was available from the HBX User Agent Filtering option. 

19 Comments (19 New)
19 Comments

Avatar

Avatar

adamgreco0

Avatar

adamgreco0

adamgreco0

11-02-2010

I think this should include a warning system that identifies suspicious IP Addresses (i.e. sening an enormous amount of traffic per day) and allow you to block that IP address from now on...

Avatar

Avatar

kate_rosser

Avatar

kate_rosser

kate_rosser

12-02-2010

This is a great idea. We currently do this via a VISTA rule but this is at an additional cost, I think this should come as standard within Site Catalyst.

Avatar

Avatar

nduley

Avatar

nduley

nduley

12-02-2010

What, it's not already filtered out? Hmmm...

Avatar

Avatar

endressanalytic

Avatar

endressanalytic

endressanalytic

12-02-2010

Yeah I was lead to believe spiders and bots are already automatically filtered out ... ?

Avatar

Avatar

RobertBlakeley

Avatar

RobertBlakeley

RobertBlakeley

14-02-2010

We do see bots in our reporting that apparently execute the Java Script. Not many, but they are there. At this time, they do not seriously affect our reporting.

Avatar

Avatar

terry70825

Avatar

terry70825

terry70825

15-02-2010

Most bots/spiders do not execute Javascript.  However, I was also under the notion that Omniture automatically handled the remaining ones that do.

Avatar

Avatar

mubarakd5726232

Avatar

mubarakd5726232

mubarakd5726232

15-02-2010

Have had this discussion/debate with omniture.  They SHOULD be filtering out sipders without cost.  One of the most important aspects of analytics is it's reliability, if bots/spiders are executing javascript or making image request (some implementation include this to capture non javascript users) then this places a big question mark over the numbers and requires further auditing.  We have in the past spotted/identified bots/spiders executing javascript and/or making image requests, doesn't really matter what the % of bots/spiders that do this might be as one hit by any of these would skew up the data.

Avatar

Avatar

jhodson

Avatar

jhodson

jhodson

15-02-2010

I have to agree that this is a big issue and an easy one to solve for Omniture.  THey should simply maintain a list of known bot IPs and have a setting in the admin consule where we can click to have this IP filter applied to our data like a standard Vista Rule.  And just like Vista rules; they would still get the server calls since it did hit their servers but we won't see the numbers in our data sets.  That's a win / win for both parties.  Clean data for us continued revenue for Omniture.

Avatar

Avatar

omuser_ideas

Avatar

omuser_ideas

omuser_ideas

16-02-2010

Bot/ spider filtering is critical for data accuracy, but we found out the hard way that this is not being done by Omniture.  The cost to implement this via a Vista rule was also a surprise.  

 

We requested that Omniture recreate the filter that was previously available through HBX, to no avail.  In the end, we had to supply the bot/ spider list ourselves.  They made a short list available, but it was incomplete and was not being updated.

 

Omniture should be using the IAB spider/ bot list (with regular updates) and implement this for all clients automatically. 

Avatar

Avatar

jamest0

Avatar

jamest0

jamest0

17-03-2010

Omniture can do this using a Vista rule, it is just a mnual task that needs to be done monthly. They can either use the IAB list of bots/spiders or in the UK ABCe publish a list of these. We are currently testing this facility which I believe should be offered to all customers as a matter of course - out of the box with no additional charge. Without credible data any insight is meaningless.