Bot traffic filtering query | Community
Skip to main content
ruchib3145524
ruchib3145524
June 7, 2019
Solved

Bot traffic filtering query

  • June 7, 2019
  • 2 replies
  • 3050 views

We are trying to analyse our previous data to filter out bots, got a list of IPs that could be potential ones.

we were planning to configure it as bot filter section.

But what if these IP does not remain bots in the future?

what are the chances? How often can an IP change?

    This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.
    Best answer by David_Jerome

    I would be interested to see how others deal with this also.

    As a travel company, we have a large number of bots / competitors that are price scraping our site. In order to maximise compatibility between Adobe Tools (Workspace / Reports / Adhoc / Data warehouse etc) we have created a Virtual Report Suite that filters out as much traffic as possible. However, we do not actually filter on IP but rather on domain, Using this technique we have found that what we see is troublesome domains - rather than IP addresses.

    Our main check is that we look at domains where we have >100 unique visitors in a week with a visits per visitor ratio of 1.0 and a conversion rate of 0%. This has also highlighted a few things to us:

    User agent strings are often a good indicator of a bot

    Adobes domain list is not awful but clearly not as as up to date as others

    Query string parameters can also help identify a bot if they are manipulating on site search.

    As for syncing this retrospectively with our data lake, well that's another problem :-/

    Thanks

    Dave

    2 replies

    David_Jerome
    David_JeromeAccepted solution
    June 7, 2019

    I would be interested to see how others deal with this also.

    As a travel company, we have a large number of bots / competitors that are price scraping our site. In order to maximise compatibility between Adobe Tools (Workspace / Reports / Adhoc / Data warehouse etc) we have created a Virtual Report Suite that filters out as much traffic as possible. However, we do not actually filter on IP but rather on domain, Using this technique we have found that what we see is troublesome domains - rather than IP addresses.

    Our main check is that we look at domains where we have >100 unique visitors in a week with a visits per visitor ratio of 1.0 and a conversion rate of 0%. This has also highlighted a few things to us:

    User agent strings are often a good indicator of a bot

    Adobes domain list is not awful but clearly not as as up to date as others

    Query string parameters can also help identify a bot if they are manipulating on site search.

    As for syncing this retrospectively with our data lake, well that's another problem :-/

    Thanks

    Dave

      V
      VikramSh
      April 6, 2024

      I believe the only solution could be user-agent and IP Filteres (IP Range).

       

      I understand your concern that in future the IP range might not signify Bots anymore but that's too minuscule of a probability to be worried about. They need to become non-bot and also become customers for your worry to become true.

       

      I do not agree that Query String parameters can be used as they are majorly caused by our own settings and scripts. therefore identification using query string parameters is not as reliable.

      Terms & ConditionsAccessibility statement

      Loading footer...