Gen AI scraper bots

Many content-rich websites with extensive knowledge bases are experiencing a significant increase in traffic from AI scrapers, crawlers, and data harvesting bots. In our case, this has been an ongoing challenge for several months. While the impact on reporting accuracy is already substantial, the more immediate concern is the associated increase in Adobe Analytics server call consumption, which can lead to unexpected overages and additional costs.

What makes this challenge different from traditional bot traffic is that these are not a small set of well-known bots that can be managed through static signatures or conventional bot rules. The traffic is highly distributed, frequently changes behavior, and often originates from a large number of sources, making it significantly harder for customers to identify and mitigate on their own.

Given the broad impact across digital analytics implementations, could Adobe consider a more proactive partnership model with customers? For example:

• Maintaining and publishing regularly updated AI scraper/bot signatures or detection patterns.
• Providing monthly or near real-time updates that customers can leverage at the network, CDN, WAF, tag management, or Analytics processing level.
• Offering enhanced detection capabilities within Adobe Analytics that specifically address emerging AI crawler traffic patterns.
• Creating a community-driven process where customers can contribute observed signatures and suspicious traffic sources that Adobe can validate and distribute more broadly.

Without stronger ecosystem-level support, organizations risk both increasing operating costs and declining confidence in the accuracy of their analytics data. As AI-driven crawlers continue to evolve, an industry-wide challenge requires a more adaptive approach than the traditional bot management strategies that were effective in the past.

I would be interested to hear whether Adobe is considering any roadmap investments or customer collaboration initiatives in this area.