Bot Traffic Through Raw Full URL | Community
Skip to main content
skatofiabah
skatofiabah
Level 5
October 23, 2024
Question

Bot Traffic Through Raw Full URL

  • October 23, 2024
  • 2 replies
  • 2169 views

Hi All,

 

We have a system where we look at the Raw Full URL of the page, then the Operating System, then the IP Address in Adobe Analytics to identify bot traffic. We are getting a full URL with "?test=1" in the query string parameter that seems to be bot traffic. Are there other ways or query strings in the full raw URL that would indicate Bot Traffic too?

 

Thanks!

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.

2 replies

PratheepArunRaj
Community Advisor and Adobe Champion
PratheepArunRajCommunity Advisor and Adobe Champion
Community Advisor and Adobe Champion
October 23, 2024

Dear @skatofiabah ,

'?test=1' need not be a bot ideally; rather, somebody is testing the page load in production (it might be your QC team) or trying to clear Akamai cache (usually, we use the query string parameter). 

Pretty not sure that using query string parameter is a reliable method to identify BOT.

Thank You, Pratheep Arun Raj B (Arun) | NextRow Digital | Terryn Winter Analytics

Thank You, Pratheep Arun Raj B (Arun) | Xerago | Terryn Winter Analytics
    V
    Vernon_H
    Level 2
    October 24, 2024

    Hi @skatofiabah,

     

    Agreed that the query string parameter would not be indicative of a bot. Personally, what we use is a mixture of User Agent, IP address, Countries, Click behavioural data, Timestamp and page views events to help identify the larger bots. I would also check the IAB filter to ensure that it is turned on (though it doesn't capture new/smaller bots).

    Cheers,

    Vernon

      skatofiabah
      skatofiabahAuthor
      Level 5
      October 24, 2024

      Hi @vernon_h,

      What is User Agent? I don't see that dimension in Adobe. Are there any other ways to slice besides those dimensions above?

      V
      Vernon_H
      Level 2
      October 25, 2024

      Hi @skatofiabah,

      For more information on user agent, here's a documentation of the string format.

      We capture user agent as a seperate eVar via the JS variable navigator.userAgent. I would personally highly recommend having this captured (especially with the rise of crawlers to power Gen AI tools). 
      An example of an infamous bot from ByteDance that appeared on our radar earlier this year:  
      mozilla/5.0 (linux; android 5.0) applewebkit/537.36 (khtml, like gecko) mobile safari/537.36 (compatible; bytespider; spider-feedback@bytedance.com) 

      From the user agent alone, you can easily do a Google search or immediately tell its some sort of a spider from bytedance.com. So you can then do an exclusion in your reports for this. 
      Other smaller scale bots might use legit user agents, hence, you will need other dimensions and metrics to support your exclusion (e.g. link clicks actions, bounce rates, time spent on page, etc.).

      Hope this info helps! Goodluck!

       

      Cheers,

      Vernon

        Terms & ConditionsAccessibility statement

        Loading footer...