Your achievements

Level 1

0% to

Level 2

Tip /
Sign in

Sign in to Community

to gain points, level up, and earn exciting badges like the new
Bedrock Mission!

Learn more

View all

Sign in to view all badges

Bot Filtering Enhancements

Avatar

Avatar
Give Back 5
Level 3
eric_matisoff
Level 3

Likes

39 likes

Total Posts

26 posts

Correct reply

1 solution
Top badges earned
Give Back 5
Give Back 3
Give Back
Boost 5
Boost 3
View profile

Avatar
Give Back 5
Level 3
eric_matisoff
Level 3

Likes

39 likes

Total Posts

26 posts

Correct reply

1 solution
Top badges earned
Give Back 5
Give Back 3
Give Back
Boost 5
Boost 3
View profile
eric_matisoff
Level 3

04-09-2015

Let's face it - bots are getting smarter. Bot Filtering needs to get smarter too. Bots are getting better at changing user-agents, IP addresses, domains and more in between visits. Bots are now even adding delays between page views to look more like a normal visitor. And don't forget, they're running JavaScript too.

 

Adobe needs to expand Bot Filtering to use look-alike modelling to discover visits with abnormally high page views.

Similar to this Idea: http://ideas.omniture.com/t5/Idea-Exchange-for-Adobe/IAB-Bot-List-Segment-in-Analytics/idi -p/11708

    we need a way to be able to apply 'Bot Segments' to reports so we can better understand what traffic changes will result prior to committing them as rules.

 

Right now the industry is losing too much accuracy in our data to rely solely on a black box of IP and user-agent based filtering.

4 Comments

Avatar

Avatar
Give Back 3
Level 9
MichaelWon
Level 9

Likes

446 likes

Total Posts

91 posts

Correct reply

1 solution
Top badges earned
Give Back 3
Ignite 5
Ignite 3
Give Back
Ignite 1
View profile

Avatar
Give Back 3
Level 9
MichaelWon
Level 9

Likes

446 likes

Total Posts

91 posts

Correct reply

1 solution
Top badges earned
Give Back 3
Ignite 5
Ignite 3
Give Back
Ignite 1
View profile
MichaelWon
Level 9

07-09-2015

agreed.  bot traffic is a huge portion (40%) of my page views right now.  I would also add that Adobe needs to give us a way to remove (or put somewhere else) this traffic from our reporting.  The bots coming to my site are changing IP addresses every few minutes which makes it impossible to exclude.

Avatar

Avatar
Give Back 5
Employee
trevorpaulsen
Employee

Likes

0 likes

Total Posts

43 posts

Correct reply

0 solutions
Top badges earned
Give Back 5
Give Back 3
Give Back
Ignite 1
View profile

Avatar
Give Back 5
Employee
trevorpaulsen
Employee

Likes

0 likes

Total Posts

43 posts

Correct reply

0 solutions
Top badges earned
Give Back 5
Give Back 3
Give Back
Ignite 1
View profile
trevorpaulsen
Employee

08-09-2015

I think a bots segment is a very clever idea.  How are you guys identifying the bots that Analytics is not currently picking up?  How do you know when something is a bot and not normal traffic when IP address, user agent, etc. are constantly changing?

Avatar

Avatar
Give Back 25
Level 5
mubarakd5726232
Level 5

Likes

88 likes

Total Posts

90 posts

Correct reply

3 solutions
Top badges earned
Give Back 25
Ignite 5
Ignite 3
Contributor
Shape 25
View profile

Avatar
Give Back 25
Level 5
mubarakd5726232
Level 5

Likes

88 likes

Total Posts

90 posts

Correct reply

3 solutions
Top badges earned
Give Back 25
Ignite 5
Ignite 3
Contributor
Shape 25
View profile
mubarakd5726232
Level 5

10-09-2015

This topic keeps coming up and I've discussed with Adobe a number of times. For now, this is what I have in place which *may* help some

 

I have two Segments..

 

-----------

SEGMENT 1

Name - Bot Review : 1PV/Visit

VISIT

Referring Domain does not exist
- AND -
Visit Depth is less than 2
- AND -
Visit Number equals 1
- AND -
Entry Featured Content (prop43) does not exist // < this is a ref appended onto some of our content, content that's either promoted internally on site or externally.

-------------

SEGMENT 2

Name : Bot Review : +50 PVs/Visit // < adjust this to what you think is well above ave

VISIT
Page Views Event is greater than 51

-------------

 

For each I have a uniques % change alert set up for different time period, ie yesterday, week

 

When the alerts are triggered, I then apply the segment on which the alert occured and then run visitor profile > domains and trend.

 

in 'most' cases it should be obvious just by looking at the domain name ie hosting companies ..such as xlhost, softlayer.com etc.  Occasionally you may have to run further reports/checking to verify eg. create a futher segment for domain, extract IPs using DW and check UA stings in logs or request pattern.

 

Once you've confirmed you can remove easily using bot filtering however this is not retrospective (another annoyance)

Avatar

Avatar
Give Back 1000
Level 10
jantzen_belliston-Adobe
Level 10

Likes

359 likes

Total Posts

2,372 posts

Correct reply

823 solutions
Top badges earned
Give Back 1000
Give back 900
Give Back 800
Give Back 700
Give back 600
View profile

Avatar
Give Back 1000
Level 10
jantzen_belliston-Adobe
Level 10

Likes

359 likes

Total Posts

2,372 posts

Correct reply

823 solutions
Top badges earned
Give Back 1000
Give back 900
Give Back 800
Give Back 700
Give back 600
View profile
jantzen_belliston-Adobe
Level 10

27-10-2020

 
Status changed to: Archived