Community Advisor and Adobe Champion

Solved

Allow Chatgpt Bot

Forum|Forum|11 months ago
March 11, 2025
3 replies
1475 views

Hi all,

does anyone know whether ChatGPT and other AI crawlers are currently already included in the IAB bot list?

We're currently having the IAB bot list feature enabled, and are capturing the user agent of incoming traffic, and as far as I can see there are not "openai" user agents captured

https://platform.openai.com/docs/bots/

Just wondering since our business stakeholders would be interested in getting the bot crawler traffic through, and the WAF is already opened up to let that kind of traffic through.

Is there actually a way to unblock a certain browser in the Bot rules that may be included in the bot list?

Best answer by FarazHusain

Hey @bjoern__koth As of now, there is no public information confirming that OpenAI's crawlers, such as GPTBot or OAI-SearchBot, are included in the IAB/ABC list, so difficult to comment.

Regarding unblocking, I doubt that's possible using bot rules, not sure if VISTA could help here.

FarazHusain

Accepted solution

Adobe Employee

Hey @bjoern__koth As of now, there is no public information confirming that OpenAI's crawlers, such as GPTBot or OAI-SearchBot, are included in the IAB/ABC list, so difficult to comment.

Regarding unblocking, I doubt that's possible using bot rules, not sure if VISTA could help here.

bjoern__koth

Author

Community Advisor and Adobe Champion

Hmm ok, so a bit of a black box. Fair enough, I cannot see any user agent containing "openai", also no Bot Name over the past 90 days.

So, maybe it's already on the IAB list

Cheers from Switzerland!

Jennifer_Dungan

Community Advisor and Adobe Champion

Hi @bjoern__koth,

If you are using the IAB list, and openai is on the list, I would expect to see it in the table you made...

On the other hand, if my site,I look at referrers for "openai" I see regular PVs and Occurrences:

So I would say that it doesn't look like they are yet in the IAB list.

Jennifer_Dungan

Community Advisor and Adobe Champion

I also am not sure about whether the IAB included AI Bots... but I am pretty sure that there is no "bot whitelist".

That said, how deep do you need the bot information? Workspace does have some limited data from Bots, including the Bot Name and the Page that the Bot hit...

So, you should be able to get a simple report without opening up traffic collection to those bots....

K

kayzee

Level 1

I would echo bjoern's post.

Knowing the content consumption by GenAI-platforms is highly valuable to evaluate since there is a behavior shift from search engines towards chat platforms.

Especially when companies like OpenAI explicitly declare their useragents.

We are able to see those useragent in server logs but I dont see them in AA too.

Additionally, I couldn't find any into if the IAB lot list contains chatgpt-like useragents/.

K

kayzee

Level 1

Just wondering if anyone has figured this out.

While tracking referral traffic from AI chatbot sites is fairly straightforward, I'm still trying to determine how to track AI bots that fetch information directly from websites to generate responses.

OpenAI has shared more details about ChatGPT bots here:
https://platform.openai.com/docs/bots

It would be great to hear Adobe’s point of view on this.

Sign up

Login with SSO

Login to the community

Login with SSO

Scanning file for viruses.

This file cannot be downloaded