Bot Filtering Enhancements

Avatar

Avatar

eric_matisoff

Avatar

eric_matisoff

eric_matisoff

04-09-2015

Let's face it - bots are getting smarter. Bot Filtering needs to get smarter too. Bots are getting better at changing user-agents, IP addresses, domains and more in between visits. Bots are now even adding delays between page views to look more like a normal visitor. And don't forget, they're running JavaScript too.

 

Adobe needs to expand Bot Filtering to use look-alike modelling to discover visits with abnormally high page views.

Similar to this Idea: http://ideas.omniture.com/t5/Idea-Exchange-for-Adobe/IAB-Bot-List-Segment-in-Analytics/idi -p/11708

    we need a way to be able to apply 'Bot Segments' to reports so we can better understand what traffic changes will result prior to committing them as rules.

 

Right now the industry is losing too much accuracy in our data to rely solely on a black box of IP and user-agent based filtering.

4 Comments

Avatar

Avatar

MichaelWon

MVP

Avatar

MichaelWon

MVP

MichaelWon
MVP

07-09-2015

agreed.  bot traffic is a huge portion (40%) of my page views right now.  I would also add that Adobe needs to give us a way to remove (or put somewhere else) this traffic from our reporting.  The bots coming to my site are changing IP addresses every few minutes which makes it impossible to exclude.

Avatar

Avatar

admoseremic

Employee

Avatar

admoseremic

Employee

admoseremic
Employee

08-09-2015

I think a bots segment is a very clever idea.  How are you guys identifying the bots that Analytics is not currently picking up?  How do you know when something is a bot and not normal traffic when IP address, user agent, etc. are constantly changing?

Avatar

Avatar

mubarakd5726232

Avatar

mubarakd5726232

mubarakd5726232

10-09-2015

This topic keeps coming up and I've discussed with Adobe a number of times. For now, this is what I have in place which *may* help some

 

I have two Segments..

 

-----------

SEGMENT 1

Name - Bot Review : 1PV/Visit

VISIT

Referring Domain does not exist
- AND -
Visit Depth is less than 2
- AND -
Visit Number equals 1
- AND -
Entry Featured Content (prop43) does not exist // < this is a ref appended onto some of our content, content that's either promoted internally on site or externally.

-------------

SEGMENT 2

Name : Bot Review : +50 PVs/Visit // < adjust this to what you think is well above ave

VISIT
Page Views Event is greater than 51

-------------

 

For each I have a uniques % change alert set up for different time period, ie yesterday, week

 

When the alerts are triggered, I then apply the segment on which the alert occured and then run visitor profile > domains and trend.

 

in 'most' cases it should be obvious just by looking at the domain name ie hosting companies ..such as xlhost, softlayer.com etc.  Occasionally you may have to run further reports/checking to verify eg. create a futher segment for domain, extract IPs using DW and check UA stings in logs or request pattern.

 

Once you've confirmed you can remove easily using bot filtering however this is not retrospective (another annoyance)

Avatar

Avatar

jantzen_belliston-Adobe

Community Manager

Avatar

jantzen_belliston-Adobe

Community Manager

jantzen_belliston-Adobe
Community Manager

27-10-2020

 
Status changed to: Archived