Highlighted

Robots.txt in not working in Publish instance

Avatar

Avatar

lovepreetk75597

Avatar

lovepreetk75597

lovepreetk75597

16-01-2020

Hi,

 

We have one requirement where we need to add robots.txt file into AEM to exclude our site from google search engine.. We have added the file on author and its working there but not working for publish instance.I have taken same package from author and import to publish instance but on publish we are getting 404 while trying to hit robots.txt directly through URL. Any help will be appreciated. Thanks in advance.

 

Regards,

Lovepreet 

Replies

Highlighted

Avatar

Avatar

Arun_Patidar

MVP

Total Posts

2.9K

Likes

1.0K

Correct Answer

831

Avatar

Arun_Patidar

MVP

Total Posts

2.9K

Likes

1.0K

Correct Answer

831
Arun_Patidar
MVP

16-01-2020

Are you accessing this vai dispatcher? Make sure .txt extension is enabled.

Highlighted

Avatar

Avatar

jbrar

Employee

Avatar

jbrar

Employee

jbrar
Employee

16-01-2020

In addition to what Arun mentioned, If you are getting 404 for the robot.txt file when directly accessing it via publisher(Bypassing dispatcher), make sure the anonymous user has access to the location of robot.txt file.

Highlighted

Avatar

Avatar

abhishekk962745

Avatar

abhishekk962745

abhishekk962745

16-01-2020

If you are on  6.4 and higher, make sure you add your domain (www.xyz.com) into your vhosts.any file under /etc/httpd/conf.dispatcher.d/vhosts

Highlighted

Avatar

Avatar

amoghd60518652

Avatar

amoghd60518652

amoghd60518652

25-01-2020

Where that file is hosted? If it is under /content/you-project then try clearing your domain cache or if it is hosted under any specific restricted path then check the anonymous access. 

Thanks,
Amogh