Identify Bots

[skatofiabah]

Hi,

How can we easily identify bots? We have a lot of bot activity happening and we need to whitelist them for both our website and Adobe Workspace and Downstream. What is a good approach to doing this?

Thanks!

[Jennifer_Dungan]

There is no surefire way to identify bots with 100% accuracy.

I am not sure what you mean by whitelist the bots? There is no inherent, automatic bot blocking (aside from the IAB bot settings that you can enable on your suite... but event that isn't automatic... that is a service that keeps an updated list of known "good bots" like Google, Bing, Yahoo, etc search engine crawlers based on the User Agents)... most people are trying to find ways to block the bots from appearing in their data.

Do you mean that you need to blacklist the bots?

Often the bad bots are something you need to monitor... either you look at identifiable user agents, or possibly suspect IP addresses or ranges where you have unusual activity (both of which you can create custom blocking rules in your "Bot" blocking settings). Then there is the segment approach for stuff that you can't create a blocking rule for... often based on traffic that is really old browsers / operating systems, or gets identified as "unknown".

[christel_guidon]

In case this helps, we covered this at our last Adobe Analytics office hours

This will soon be posted on the Adobe Analytics User Group Page under Office Hours: https://www.youtube.com/channel/UCQOHnCs7KZgsuFHVzwboQuA/featured 

Other champions had great tips, we covered that about half way through the session (not sure?)

Until then, here are my notes on the subject in case it helps

Admin out of the box settings

• Go to Admin > Report suites > select report suite > edit settings > General menu > Bot rules
• Enable IAB Bot Filtering Rules (checkbox)
• You can setup your own bot rules as well (csv import or one-offs) using user agent, ip address, ip ranges. There are some options there (starts with, contains, etc....)

Adobe Bot reports

• Adobe has a Bot Name dimension that can be used with the Bot Page Views, Bot Occurrences and Bot Page View Ratio metrics (only)
• These will have a warning attached to them that you can not use them with other dimensions and metrics
• Cannot be used prior to Feb/Mar 2023
• As part of my 'Monitor Dashboard', I monitor:
• The page view ratio overall
• Bot page views and occurrences per day
• Trend of top 5 Bots
• I have alerts based on these as well

Capture user agents using a processing rule

• Go to Admin > Report suites > select report suite > edit settings > General menu > Processing rules
• Dedicate a new evar to capture User Agents
• If Adobe Visitor ID is not set, write user agent in this new evar (I have an evar that captures the Adobe MCID, that is what I use for this condition)
• I would also add a condition that says 'if visitor id not set AND another evar that is always set on your pages is not set, then put user agent in v80'. I use my property tag for the evar that is always set
• After setting this rule up, you can monitor your user agents and get alerted if you see any spikes. I also monitor this as part of my 'Monitor Dashboard'
• When something looks off, you can add it to the rules in the Admin settings

Set up VISTA rules with Adobe (cost)

• Adobe can set up VISTA rules for you that can exclude certain user agents
• These need to be based on a specific dimension and come at an additional cost
• They can create more complex options for you than what is available in the admin settings
• A DB Vista rule (vs a 'normal' Vista rule) will also allow you to modify and add to the rule yourself, but is more expensive