Preventing server calls from bot traffic

Question

Hi,

We have an Adobe Experience Platform report suit that is currently receiving server calls from know bot user agents. Bot filtering prevents these coming into our front end reporting.

Following Adobe documentation, we have updated JS code to exclude user agents containing 'Googlebot' using the onBeforeEventSend callback, however we are continuing to receive server calls.

Please could you review the code and advise if there are any amendments or additions to be made?

Thanks

Neil

Jennifer_Dungan · Accepted Answer

Yes, "WebSDK" is not to be confused with using Adobe Launch (err... Adobe Data Collection).

AppMeasurement.js is the client side, "old school" way of tracking, Alloy.js / WebSDK is the server side tracking that relies on the XDM Stream. Both solutions can be controlled via Adobe's tag manager (Launch), or in GTM, or even deployed directly into the site code.

However, now that I know you are not using Launch, the "conditions" that I mentioned wouldn't work, but you should be able to do something similar with GTM (adding constraints onto the Page Views trigger), if you really want to stop tracking calls made via bots... the principal is essentially the same, add conditions to your rules rather than trying to abort the call at the end...

I would still think that the onBeforeEventSend would be in both page view and action calls though... I wonder if you could add some sort of traceability (like adding console.logs inside the onBeforeEventSend function (to catch when that triggers), then again inside the navigator logic to catch when "Googlebot" is identified...

Something like:

onBeforeEventSend: function() {
    console.log("!!!!!!!!!!!!!! onBeforeEventSend triggered");
    console.log("!!!!!!!!!!!!!! " + navigator.userAgent);
    if (navigator && navigator.userAgent && navigator.userAgent.includes("Googlebot")) {
        console.log("!!!!!!!!!!!!!! Googlebot identified");
        return false;
    }
}

You can pretend to be Googlebot through the use of some "User Agent Switcher" add-ons for your browser... you can then test this and see what is happening.

If this looks like it might be working locally, then you might want to consider setting up some new dimensions to track the userAgent and other information that might help you track down what calls are not being caught by this logic.

Good luck!

Jennifer_Dungan · Answer

Hmmm the code to identify Googlebot looks solid, I am not sure about the onBeforeEventSend function though... I am not using WebSDK yet, so I can't check this...

But, as an alternate suggestion, and possibly one that might also make your site more efficient.. have you considered instead of trying to abort the call before sending (meaning all your rule logic is still running), what about using a condition on your rules... if the user agent is Googlebot (or other identified bot), don't even run the rule, which then in turn shouldn't trigger the tracking call at all?

if (navigator && navigator.userAgent && navigator.userAgent.includes("Googlebot")){
  return true;
}

Basically, if the user agent contains Googlebot, return true, and since we are using an "Exception" rule, this should prevent the rule from running completely.

You would have to apply this to all rules, but it also means that if it's Googlebot, you aren't running all the JS to build out your tracking request....

Sign up

Login with SSO

Login to the community

Login with SSO

Scanning file for viruses.

This file cannot be downloaded