LLM txt file is not accessible by gemini or any gpt | Community
Skip to main content
veerareddyc1015
veerareddyc1015
Level 3
June 8, 2026
Question

LLM txt file is not accessible by gemini or any gpt

  • June 8, 2026
  • 3 replies
  • 80 views

Hi Team,

 

Could you please provide your guidance on the following?

We recently created an llms.txt file under DAM with some content in it. This file is accessible via the domain (for example: https://www.mycompany.com/llms.txt). URL shortening is being handled through the dispatcher, and the file loads successfully in the browser without any issues.

 

However, when we try to access the same URL through Gemini or any gpt and request it to read the content, it returns an error stating that the file is not accessible or not available on the domain.

 

Since we have Microsoft CDN and AMS WAF rules in place, could you please advise whether we need to validate this at the CDN layer, AMS App Gateway WAF rules, or the dispatcher level?

 

Thanks in Advance

Regrads

Veera

 

3 replies

narendragandhi
Community Advisor
narendragandhiCommunity Advisor
Community Advisor
June 9, 2026

Hi ​@veerareddyc1015 

 

You can refer this - https://scrunch.com/resources/guides/guide-to-ai-user-agents/  to verify if these user agents are whitelisted to allow them to connect to your website.

 

Thanks
Narendra

    Nilesh_Mali
    Nilesh_Mali
    Level 3
    June 10, 2026

    Hi ​@veerareddyc1015 

    If the file is accessible directly in a browser but not by Gemini, ChatGPT, or other AI crawlers, the issue is likely at the CDN/WAF layer rather than AEM itself.

    Since the URL is being rewritten through the CDN/Dispatcher, ensure that the CDN behavior explicitly allows access to the llms.txt file and does not block requests from AI crawler user agents. Whitelisting the specific file path at the CDN/WAF level is usually required.

    Also, consider publishing additional AI-friendly metadata files such as agents.md, skills.md, index.md, and architecture.md. These files can provide structured information about your site, capabilities, and content architecture, which may be more useful for AI agents than relying solely on a single llms.txt file. Also this .md files does not require whitelisting. 

    Recommended validation points:

    • CDN rules and behaviour's
    • WAF/App Gateway policies
    • Dispatcher filters and cache rules
    • Response headers and bot access restrictions
    avesh_narang
    avesh_narang
    Level 4
    June 14, 2026

    Hi ​@veerareddyc1015  since browser access is working, I would request to first check CDN and WAF logs, followed by validation of dispatcher filters and response headers.

    The most likely cause is that AI crawler user agents are being blocked at the CDN/WAF layer.

    But again logs will help here, if feasible please share logs .

     

    Thanks 

      Terms & ConditionsAccessibility statement

      Loading footer...