Skip to main content
shrutid56001193
User Group Leader
shrutid56001193User Group Leader
User Group Leader
Jun 18, 2025 11:13 AM UTC

AEMaaCS Robots.txt points to to default Adobe one, not the custom one

Hello,

 

We are implementing custom robots.txt which will be maintainable by content authors. the file is located in: "/content/dam/mycompany/config/robots.txt"

 

We implemented following Dispatcher changes and deployed to AEMaaCS env:

 

1. filters.any

# Allow robots.txt /0104 { /type "allow" /url "/content/dam/mycompany/config/robots.txt" /extension "txt"} /0105 { /type "allow" /url "/robots.txt" /extension "txt"}

 

2. To open file inline, rather than downloading it: in dispatcher/src/conf.d/available_vhosts

<LocationMatch "^/content/.*/robots\.txt$"> Header unset "Content-Disposition" Header set Content-Disposition inline </LocationMatch

 

3. rewrite.rules:

RewriteCond %{REQUEST_URI} ^/robots\.txt$ RewriteRule .* /content/dam/mycompany/config/robots.txt [PT,L]

 

Still, the Adobe default robots.file is shown, not our custom one.

what seems to be missing?

 

Appreciate your help!

Regards,

Shruti

 

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.

10 replies

Saravanan_Dharmaraj
Community Advisor
Saravanan_DharmarajCommunity Advisor
Community Advisor
Jun 18, 2025 4:32 PM UTC

@shrutid56001193  When you directly hit the dispatcher URL http://<disp_domain>/robots.txt , does it pick up to your file? check the dispatcher logs too. If hit is processed correctly in dispatcher then the issue might be with CDN , check if there is any redirect there.  

Translate
    shrutid56001193
    User Group Leader
    shrutid56001193User Group LeaderAuthor
    User Group Leader
    Jun 19, 2025 7:17 AM UTC

    thanks, yes this seems like cached at CDN level, this is AEM as a Cloud service and we are using Adobe bundled fastly CDN. so when i try publish+dispatcher url, its always with CDN. how to bypass CDN? 

    Translate
    https://www.shrutidamle.com/
    konstantyn_diachenko
    Community Advisor
    konstantyn_diachenkoCommunity Advisor
    Community Advisor
    Jun 19, 2025 4:19 PM UTC

    Hi @shrutid56001193 ,

     

    For testing purpose try to change your rewrite rule to allow query parameters:

    RewriteCond %{REQUEST_URI} ^/robots\.txt RewriteRule .* /content/dam/mycompany/config/robots.txt [PT,L]

     

    To bypass CDN cache, try to call /robots.txt?nocache=1.

    Translate
    Kostiantyn Diachenko, Community Advisor, Certified Senior AEM Developer, creator of free AEM VLT Tool, maintainer of AEM Tools plugin.
      konstantyn_diachenko
      Community Advisor
      konstantyn_diachenkoCommunity Advisor
      Community Advisor
      Jun 18, 2025 5:38 PM UTC

      Hi @shrutid56001193 ,

       

      Try to replace your dispatcher filter with the next one:

      # Allow robots.txt /0105 { /type "allow" /method "GET" /path "/content/dam/mycompany/config/robots" /extension "txt" }

       

      Translate
      Kostiantyn Diachenko, Community Advisor, Certified Senior AEM Developer, creator of free AEM VLT Tool, maintainer of AEM Tools plugin.
        shrutid56001193
        User Group Leader
        shrutid56001193User Group LeaderAuthor
        User Group Leader
        Jun 19, 2025 10:27 AM UTC

        thanks @konstantyn_diachenko , but it didn't work...

        Translate
        https://www.shrutidamle.com/
        Shashi_Mulugu
        Community Advisor and User Group Leader
        Shashi_MuluguCommunity Advisor and User Group Leader
        Community Advisor and User Group Leader
        Jun 19, 2025 4:58 AM UTC

        @shrutid56001193  Pls try @konstantyn_diachenko solution,  also try to clear dispatcher cache after any rules change,  you might be serving already cached file..

         

        If it doesn't work, start hitting url with publisher directly,  check if dispatcher calling publisher? If yes with what part etc..

        Translate
        shrutid56001193
        User Group Leader
        shrutid56001193User Group LeaderAuthor
        User Group Leader
        Jun 19, 2025 1:03 PM UTC

        its AEM as a cloud service, so not possible to hit the publisher directly.

        Translate
        https://www.shrutidamle.com/
        Luis1782324
        Oct 31, 2025 12:26 PM UTC

        Hi @shrutid56001193 , did you find a solution?

        Translate
        shrutid56001193
        User Group Leader
        shrutid56001193User Group LeaderAuthor
        User Group Leader
        Oct 31, 2025 12:49 PM UTC

        There is no clear solution but 2 workarounds:

         

        Workaround 1:

        Only for dev and stage domains, we can use custom path for robots.txt, e.g. https://publish-pxxxx-exxxx.adobeaemcloud.com/my-path/robots.txt  - which shows custom robots file.

        Workaround 2:

        Configure custom domain names for dev and stage envs - then /robots txt will point to the custom robots file.

         

        We went ahead with Workaround 1: as this is only for dev, stage. not prod.

        Translate
        https://www.shrutidamle.com/
        Terms & ConditionsAccessibility statement

        Loading footer...