Robots.txt file isn't accessible on publish environment AEM as a cloud service | Community
Skip to main content
JeffreyVSpringbok
JeffreyVSpringbok
Level 2
March 22, 2023

Robots.txt file isn't accessible on publish environment AEM as a cloud service

  • March 22, 2023
  • 1 reply
  • 2003 views

Hi

 

We are implementing a robots.txt file that needs to be fetched from the dam (for several sites with several languages).

The first issue that we are facing is that we can't access the txt file on the publish environment with the direct path e.g. /content/dam/robots/<sitename>/en_gb/robots.txt.

 

When we try to put it on a link on a button same result (href attribute gets removed).
The same approach works fine with pdf files.

 

Does someone knows if we missed something regarding the filetypes that can't be published/accessed on the publish environment?

We are using AEM as a cloud service version : 2022.11.9850.20221116T162329Z.

 

 

Thanks

 

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.

1 reply

arunpatidar
Community Advisor
arunpatidarCommunity Advisor
Community Advisor
March 22, 2023

1. you need to write, redirect rule

https://github.com/arunpatidar02/aemaacs-aemlab/blob/5d778dd40cbcc899931f0570fb9b5e2a40a2d616/dispatcher.cloud/src/conf.d/rewrites/default_rewrite.rules#L12

2. allow robots.txt from dispatcher filter

 

Arun Patidar
    JeffreyVSpringbok
    JeffreyVSpringbokAuthor
    Level 2
    March 22, 2023

    Hi Arun

     

    We have enabled the right rules in the dispatcher but we still have the same issue.
    As well if I bypass the dispatcher  by accessing the publish url: 

    https://publish-pXXXXX-eXXXXX.adobeaemcloud.com/content/dam/robots/<sitename>/en_gb/robots.txt

    It still gives a 404


    If I try to access a jpg file it works.

      JeffreyVSpringbok
      JeffreyVSpringbokAuthor
      Level 2
      March 29, 2023

      Please check if this helps

      https://experienceleaguecommunities.adobe.com/t5/adobe-experience-manager/facing-issues-for-robots-txt-in-aem-cloud-sdk/m-p/410027


      Hi Arun
      Small update on my side.
      If I try it on my own local dispatcher instance it is working fine after performing the following chance.

      /0105 { /type "allow" /method "GET" /url "/robots" /extension "txt" }

      I allowed also all the .txt files coming from my content path.

      0100 { /type "allow" /extension '(css|eot|gif|ico|jpeg|jpg|js|gif|pdf|png|svg|swf|ttf|woff|woff2|html|txt)' /path "/content/*" }

      I still see in the logs actionblocked.

        Terms & ConditionsAccessibility statement

        Loading footer...