Robots.txt in not working in Publish instance | Community
Skip to main content
lovepreetk75597
lovepreetk75597
Level 2
January 16, 2020

Robots.txt in not working in Publish instance

  • January 16, 2020
  • 4 replies
  • 3738 views

Hi,

 

We have one requirement where we need to add robots.txt file into AEM to exclude our site from google search engine.. We have added the file on author and its working there but not working for publish instance.I have taken same package from author and import to publish instance but on publish we are getting 404 while trying to hit robots.txt directly through URL. Any help will be appreciated. Thanks in advance.

 

Regards,

Lovepreet 

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.

4 replies

arunpatidar
Community Advisor
arunpatidarCommunity Advisor
Community Advisor
January 16, 2020

Are you accessing this vai dispatcher? Make sure .txt extension is enabled.

Arun Patidar
    U
    Adobe Employee
    user05162Adobe Employee
    Adobe Employee
    January 16, 2020

    In addition to what Arun mentioned, If you are getting 404 for the robot.txt file when directly accessing it via publisher(Bypassing dispatcher), make sure the anonymous user has access to the location of robot.txt file.

      K
      kollurab
      Level 2
      January 16, 2020

      If you are on  6.4 and higher, make sure you add your domain (www.xyz.com) into your vhosts.any file under /etc/httpd/conf.dispatcher.d/vhosts

        A
        amogh-personal
        Level 2
        January 26, 2020

        Where that file is hosted? If it is under /content/you-project then try clearing your domain cache or if it is hosted under any specific restricted path then check the anonymous access. 

        Thanks,
        Amogh

          Terms & ConditionsAccessibility statement

          Loading footer...