Your achievements

Level 1

0% to

Level 2

Tip /
Sign in

Sign in to Community

to gain points, level up, and earn exciting badges like the new
BedrockMission!

Learn More

View all

Sign in to view all badges

Filter out Spider and Bot traffic from SiteCatalyst

Avatar

Avatar
Boost 500
Level 10
vabeachkevin1
Level 10

Likes

553 likes

Total Posts

40 posts

Correct Reply

0 solutions
Top badges earned
Boost 500
Boost 50
Boost 5
Boost 3
Boost 250
View profile

Avatar
Boost 500
Level 10
vabeachkevin1
Level 10

Likes

553 likes

Total Posts

40 posts

Correct Reply

0 solutions
Top badges earned
Boost 500
Boost 50
Boost 5
Boost 3
Boost 250
View profile
vabeachkevin1
Level 10

11-02-2010

There should be an option to filter out all spider and bot traffic from SiteCatalyst reports, very similar to what was available from the HBX User Agent Filtering option. 

19 Comments

Avatar

Avatar
Boost 500
Level 10
adamgreco0
Level 10

Likes

1,936 likes

Total Posts

123 posts

Correct Reply

0 solutions
Top badges earned
Boost 500
Boost 50
Boost 5
Boost 3
Boost 250
View profile

Avatar
Boost 500
Level 10
adamgreco0
Level 10

Likes

1,936 likes

Total Posts

123 posts

Correct Reply

0 solutions
Top badges earned
Boost 500
Boost 50
Boost 5
Boost 3
Boost 250
View profile
adamgreco0
Level 10

11-02-2010

I think this should include a warning system that identifies suspicious IP Addresses (i.e. sening an enormous amount of traffic per day) and allow you to block that IP address from now on...

Avatar

Avatar
Boost 50
Level 4
kate_rosser
Level 4

Likes

56 likes

Total Posts

3 posts

Correct Reply

0 solutions
Top badges earned
Boost 50
Boost 5
Boost 3
Boost 25
Boost 10
View profile

Avatar
Boost 50
Level 4
kate_rosser
Level 4

Likes

56 likes

Total Posts

3 posts

Correct Reply

0 solutions
Top badges earned
Boost 50
Boost 5
Boost 3
Boost 25
Boost 10
View profile
kate_rosser
Level 4

12-02-2010

This is a great idea. We currently do this via a VISTA rule but this is at an additional cost, I think this should come as standard within Site Catalyst.

Avatar

Avatar
Boost 5
Level 2
nduley
Level 2

Likes

5 likes

Total Posts

3 posts

Correct Reply

0 solutions
Top badges earned
Boost 5
Boost 3
Boost 1
Applaud 5
Applaud 25
View profile

Avatar
Boost 5
Level 2
nduley
Level 2

Likes

5 likes

Total Posts

3 posts

Correct Reply

0 solutions
Top badges earned
Boost 5
Boost 3
Boost 1
Applaud 5
Applaud 25
View profile
nduley
Level 2

12-02-2010

What, it's not already filtered out? Hmmm...

Avatar

Avatar
Level 1
endressanalytic
Level 1

Likes

0 likes

Total Posts

1 post

Correct Reply

0 solutions
View profile

Avatar
Level 1
endressanalytic
Level 1

Likes

0 likes

Total Posts

1 post

Correct Reply

0 solutions
View profile
endressanalytic
Level 1

12-02-2010

Yeah I was lead to believe spiders and bots are already automatically filtered out ... ?

Avatar

Avatar
Contributor
Level 7
RobertBlakeley
Level 7

Likes

200 likes

Total Posts

95 posts

Correct Reply

0 solutions
Top badges earned
Contributor
Shape 25
Shape 10
Shape 1
Validate 1
View profile

Avatar
Contributor
Level 7
RobertBlakeley
Level 7

Likes

200 likes

Total Posts

95 posts

Correct Reply

0 solutions
Top badges earned
Contributor
Shape 25
Shape 10
Shape 1
Validate 1
View profile
RobertBlakeley
Level 7

14-02-2010

We do see bots in our reporting that apparently execute the Java Script. Not many, but they are there. At this time, they do not seriously affect our reporting.

Avatar

Avatar
Applaud 5
Level 1
terry70825
Level 1

Likes

0 likes

Total Posts

2 posts

Correct Reply

0 solutions
Top badges earned
Applaud 5
View profile

Avatar
Applaud 5
Level 1
terry70825
Level 1

Likes

0 likes

Total Posts

2 posts

Correct Reply

0 solutions
Top badges earned
Applaud 5
View profile
terry70825
Level 1

15-02-2010

Most bots/spiders do not execute Javascript.  However, I was also under the notion that Omniture automatically handled the remaining ones that do.

Avatar

Avatar
Contributor
Level 5
mubarakd5726232
Level 5

Likes

86 likes

Total Posts

89 posts

Correct Reply

3 solutions
Top badges earned
Contributor
Shape 25
Shape 10
Shape 1
Ignite 1
View profile

Avatar
Contributor
Level 5
mubarakd5726232
Level 5

Likes

86 likes

Total Posts

89 posts

Correct Reply

3 solutions
Top badges earned
Contributor
Shape 25
Shape 10
Shape 1
Ignite 1
View profile
mubarakd5726232
Level 5

15-02-2010

Have had this discussion/debate with omniture.  They SHOULD be filtering out sipders without cost.  One of the most important aspects of analytics is it's reliability, if bots/spiders are executing javascript or making image request (some implementation include this to capture non javascript users) then this places a big question mark over the numbers and requires further auditing.  We have in the past spotted/identified bots/spiders executing javascript and/or making image requests, doesn't really matter what the % of bots/spiders that do this might be as one hit by any of these would skew up the data.

Avatar

Avatar
Applaud 5
Level 1
jhodson
Level 1

Likes

0 likes

Total Posts

1 post

Correct Reply

0 solutions
Top badges earned
Applaud 5
View profile

Avatar
Applaud 5
Level 1
jhodson
Level 1

Likes

0 likes

Total Posts

1 post

Correct Reply

0 solutions
Top badges earned
Applaud 5
View profile
jhodson
Level 1

15-02-2010

I have to agree that this is a big issue and an easy one to solve for Omniture.  THey should simply maintain a list of known bot IPs and have a setting in the admin consule where we can click to have this IP filter applied to our data like a standard Vista Rule.  And just like Vista rules; they would still get the server calls since it did hit their servers but we won't see the numbers in our data sets.  That's a win / win for both parties.  Clean data for us continued revenue for Omniture.

Avatar

Avatar
Boost 5
Level 2
omuser_ideas
Level 2

Likes

13 likes

Total Posts

5 posts

Correct Reply

0 solutions
Top badges earned
Boost 5
Boost 3
Boost 10
Boost 1
Applaud 5
View profile

Avatar
Boost 5
Level 2
omuser_ideas
Level 2

Likes

13 likes

Total Posts

5 posts

Correct Reply

0 solutions
Top badges earned
Boost 5
Boost 3
Boost 10
Boost 1
Applaud 5
View profile
omuser_ideas
Level 2

16-02-2010

Bot/ spider filtering is critical for data accuracy, but we found out the hard way that this is not being done by Omniture.  The cost to implement this via a Vista rule was also a surprise.  

 

We requested that Omniture recreate the filter that was previously available through HBX, to no avail.  In the end, we had to supply the bot/ spider list ourselves.  They made a short list available, but it was incomplete and was not being updated.

 

Omniture should be using the IAB spider/ bot list (with regular updates) and implement this for all clients automatically. 

Avatar

Avatar
Boost 5
Level 2
jamest0
Level 2

Likes

19 likes

Total Posts

13 posts

Correct Reply

0 solutions
Top badges earned
Boost 5
Boost 3
Boost 10
Boost 1
Applaud 5
View profile

Avatar
Boost 5
Level 2
jamest0
Level 2

Likes

19 likes

Total Posts

13 posts

Correct Reply

0 solutions
Top badges earned
Boost 5
Boost 3
Boost 10
Boost 1
Applaud 5
View profile
jamest0
Level 2

17-03-2010

Omniture can do this using a Vista rule, it is just a mnual task that needs to be done monthly. They can either use the IAB list of bots/spiders or in the UK ABCe publish a list of these. We are currently testing this facility which I believe should be offered to all customers as a matter of course - out of the box with no additional charge. Without credible data any insight is meaningless.