Filtering bot traffic using an evar and VRS

[Kayzee08]

Hi Crew!

I’m working on strengthening bot filtering in Adobe Analytics.
I'm able to identify and assign cookies to bot traffic (including those from the “allowed list”).
Naturally, I need to exclude this traffic from reporting (using Virtual Report Suites).
I’ve configured the system to pass the cookie value to an eVar that is set at the visit level and configured to "never expire".
Then I’m filtering it out via a segment based on the "visitor" level that excludes matching the cookie value created for bots.

Hi Crew!

I’m working on strengthening bot filtering in Adobe Analytics.
I'm able to identify and assign cookies to bot traffic (including those from the “allowed list”).
Naturally, I need to exclude this traffic from reporting (using VRS).

I’m passing the cookie value into an evar that is set to "never expire".
Then I’m using a segment at the "visitor" level to exclude users whose evar matches the cookie value associated with bots.

Does this sound like the right approach?

Don't think that setting this evar on a visitor or hit level is better since I always need to exclude this traffic.

ps: will try to get to the point where we dont even fire AA if we see the cookie earlier.

pps: I can't do bot expulsion based on user agent or IPs.

[bjoern__koth · Accepted solution]

Hi @Kayzee08 ,

you're surely on the right track! Using a segment and VRS combination, you can (and will) extend the segment definition with new findings along the way. Unfortunately, this is always a very reactive approach and potentially, data that has been reported on will be corrected through needed segment adaptations.

I would say you won't even need a cookie for that if you can identify the bot traffic throug a bot segment in AA. Looking at old browser versions with high amounts of traffic, operating systems that typically are used mainly on servers (e.g., Linux) are a very good starting point to build up your segment definition.

Quite often, if you see unusual spikes, you may see that they are also coming from certain "Cities" which should be taken into consideration as well.

Bear in mind that you can / should also enable the IAB Bot List in the bot rules. From experience, I would say that this list is only covering the "friendly bots" like search engine crawlers and custom crawlers will likely not identitfy them as such and use a modern user agent signature (i.e., will then have to be added to your segment).

[MandyGeorge]

I absolutely agree with @bjoern__koth's response.

The one thing I would add, I would be careful with using visitor level segments to exclude bot traffic. If there have been visits to the site that weren't flagged, and then one is, when it excludes that one it'll exclude all the historical visits too. So this could impact historical reporting/stuff that has already been reported on. We do all of our bot exclusions at a visit level so we don't have this issue.

[Jennifer_Dungan]

Yes, I agree with the others, Creating a flag in a dimension is a great way to identify "bad traffic", and using a Virtual Report Suite to exclude that traffic is a great way to go if you can't use the built-in Bot Exclusion Rules.

Like Bjoern said, enabling the standard IAB Bot Filtering is still a good idea to weed those calls out as well.

However, there are definitely two schools of thought on "Visitor Level Exclusion".

As Mandy said, this could affect historical data... but that may well be intentional. If historical data is inflated due to uncaught bots, then this would "right size" the data after certain traffic would be identified.... this would have to be a judgement call on if you want that particular behaviour.. and of course, you can always go and see the impact of the identified "Bot" traffic by looking at the parent suite, and using a second segment to include this data.

And because the data will still be available in the parent suite, you can create a scheduled report showing the impact of the exclusion.

Just as an FYI, Adobe also has an "Audiences" feature, where you can identify an Audience based on some criteria, and it's "Visitor" based as well... In the past (this was back when we still had Reports and Analytics), people would use the Audiences / Customer Attributes to flag bot traffic and use that to exclude identified data. With Segments, it's easier to maintain, but the point is: that also worked retroactively (removing visitors from historical data if they were flagged as bot traffic). So this approach is acceptable if you are trying to fix data going forwards, but also historically.

I have done something similar for years, not so much for Bot traffic (though I have been thinking about rolling in uncaught bot traffic to right size the data once discovered, and to fix historical data in addition to adding a bot rule), but I use it to fix issues that have come up in the past... like one time a decision was made to refresh our sites when a user changed a breakpoint (to better deal with some of the adaptive content)... however, they accidentally had an overlap on Portrait iPad screen size, which lead to those users getting stuck in a loading loop... so over the weekend, we had a huge unrealistic traffic spike on tablet, which I resolved by creating a segment for the few days of the issue and excluding "reloads" for iPads in portrait mode..

So there are lots of use cases for this approach. I wish you the best of luck.

[Kayzee08]

thank you all for sharing feedback and thoughts!
appreciate all the help!

side question - has anyone tried implementing bot exclusion on the tagging layer so if the bot can be detected way earlier then you'd avoid launching AA at all. Any cons? (we can detect it for every hit to servers)

ps: I have IAB enabled too. 

[Jennifer_Dungan]

You're welcome!

As for blocking bot traffic at the Launch level; I have not... there are a few reason for this:

1. It would kill the Bot Reports, which I also use to determine good and bad bots crawling our site, and can use this information to support DevOps

2. If something goes wrong, and traffic is excluded, it's completely lost... at least with Bot Filtering and Segments I can monitor what is happening and make adjustments if there are issues.

I get that this can be tempting to reduce server calls, but it really hinders the ability to monitor the detection rules and the ability to support server health reporting.