Allow Chatgpt Bot

[bjoern__koth]

Hi all,

does anyone know whether ChatGPT and other AI crawlers are currently already included in the IAB bot list?

We're currently having the IAB bot list feature enabled, and are capturing the user agent of incoming traffic, and as far as I can see there are not "openai" user agents captured

https://platform.openai.com/docs/bots/

Just wondering since our business stakeholders would be interested in getting the bot crawler traffic through, and the WAF is already opened up to let that kind of traffic through.

Is there actually a way to unblock a certain browser in the Bot rules that may be included in the bot list?

[FarazHusain · Accepted solution]

Hey @bjoern__koth As of now, there is no public information confirming that OpenAI's crawlers, such as GPTBot or OAI-SearchBot, are included in the IAB/ABC list, so difficult to comment. 

Regarding unblocking, I doubt that's possible using bot rules, not sure if VISTA could help here. 

[bjoern__koth]

Hmm ok, so a bit of a black box. Fair enough, I cannot see any user agent containing "openai", also no Bot Name over the past 90 days.

So, maybe it's already on the IAB list

[Jennifer_Dungan]

Hi @bjoern__koth,

If you are using the IAB list, and openai is on the list, I would expect to see it in the table you made...

On the other hand, if my site,I look at referrers for "openai" I see regular PVs and Occurrences:

So I would say that it doesn't look like they are yet in the IAB list.

[UjjwalChandra]

Hi @FarazHusain ,
Can you give some brief knowledge about VISTA? What is it used for and how?

[Jennifer_Dungan]

Hi @UjjwalChandra,

VISTA Rules are basically some extra backend rules that can be applied to your tracking. These rules have an extra cost associated to building them since they require an engineer to work with you to build them and apply them to your site.

I am not sure the full capabilities of what Vista rules can do, but I believe they can include:

• data manipulation (i.e. if dimension x is "value", then apply values or events to other dimensions/metrics - mostly what can be done with Processing Rules, but I since there is more control, you could actually split values or use more complex matching rules like Regex that you can't in Processing Rules)
• I believe you can also use them to transform and manipulate data to another tracking suite - when I was working on a migration of one outlier site to our Global Tracking Suite, I had considered using Vista Rules to transform the tracking into the older suite (putting the correct values into their dimension configuration, and manipulating the values into the formatting required by the old suite...) I ended up not doing this, I found an alternate solution....

Basically consider them to be extra SQL manipulation of the data which requires complex logic that you can't do any other way... however, due to the associated costs, it's best to investigate alternate solutions that still suite your needs first.

[FarazHusain]

Hey @UjjwalChandra , as explained by @Jennifer_Dungan, these are just some extra backend rules, but you cannot directly set up them within Adobe Analytics UI. They require additional engagement and scoping depending on your use case. In case you prefer to use them, then you can contact your Account Team.

Some common use cases of VISTA rules include:

• Copy an Analytics hit from one report suite to another, optionally altering data to the copied report suite
• Custom IP exclusion that exceeds the use cases offered by Exclude by IP
• Conditionally or globally modify any variable value
• Duplicate variable values to other variables
• Upload files to an Adobe FTP site that can impact variable values

[Jennifer_Dungan]

I also am not sure about whether the IAB included AI Bots... but I am pretty sure that there is no "bot whitelist".

That said, how deep do you need the bot information? Workspace does have some limited data from Bots, including the Bot Name and the Page that the Bot hit...

So, you should be able to get a simple report without opening up traffic collection to those bots....

[kayzee]

I would echo bjoern's post.

Knowing the content consumption by GenAI-platforms is highly valuable to evaluate since there is a behavior shift from search engines towards chat platforms.

Especially when companies like OpenAI explicitly declare their useragents.

We are able to see those useragent in server logs but I dont see them in AA too.

Additionally, I couldn't find any into if the IAB lot list contains chatgpt-like useragents/.

[kayzee]

Just wondering if anyone has figured this out.

While tracking referral traffic from AI chatbot sites is fairly straightforward, I'm still trying to determine how to track AI bots that fetch information directly from websites to generate responses.

OpenAI has shared more details about ChatGPT bots here:
https://platform.openai.com/docs/bots

It would be great to hear Adobe’s point of view on this.