Bot Filtering Enhancements

[#1]

Let's face it - bots are getting smarter. Bot Filtering needs to get smarter too. Bots are getting better at changing user-agents, IP addresses, domains and more in between visits. Bots are now even adding delays between page views to look more like a normal visitor. And don't forget, they're running JavaScript too.

Adobe needs to expand Bot Filtering to use look-alike modelling to discover visits with abnormally high page views.

Similar to this Idea: http://ideas.omniture.com/t5/Idea-Exchange-for-Adobe/IAB-Bot-List-Segment-in-Analytics/idi -p/11708

    we need a way to be able to apply 'Bot Segments' to reports so we can better understand what traffic changes will result prior to committing them as rules.

Right now the industry is losing too much accuracy in our data to rely solely on a black box of IP and user-agent based filtering.

[#2]

agreed.  bot traffic is a huge portion (40%) of my page views right now.  I would also add that Adobe needs to give us a way to remove (or put somewhere else) this traffic from our reporting.  The bots coming to my site are changing IP addresses every few minutes which makes it impossible to exclude.

[#3]

I think a bots segment is a very clever idea.  How are you guys identifying the bots that Analytics is not currently picking up?  How do you know when something is a bot and not normal traffic when IP address, user agent, etc. are constantly changing?

[#4]

This topic keeps coming up and I've discussed with Adobe a number of times. For now, this is what I have in place which *may* help some

I have two Segments..

-----------

SEGMENT 1

Name - Bot Review : 1PV/Visit

VISIT

Referring Domain does not exist
- AND -
Visit Depth is less than 2
- AND -
Visit Number equals 1
- AND -
Entry Featured Content (prop43) does not exist // < this is a ref appended onto some of our content, content that's either promoted internally on site or externally.

-------------

SEGMENT 2

Name : Bot Review : +50 PVs/Visit // < adjust this to what you think is well above ave

VISIT
Page Views Event is greater than 51

-------------

For each I have a uniques % change alert set up for different time period, ie yesterday, week

When the alerts are triggered, I then apply the segment on which the alert occured and then run visitor profile > domains and trend.

in 'most' cases it should be obvious just by looking at the domain name ie hosting companies ..such as xlhost, softlayer.com etc.  Occasionally you may have to run further reports/checking to verify eg. create a futher segment for domain, extract IPs using DW and check UA stings in logs or request pattern.

Once you've confirmed you can remove easily using bot filtering however this is not retrospective (another annoyance)