SEO | Pages on staging servers pages must be omitted from outside bots such as Google, Bing/Yahoo, Yandex etc.

[swamulu143]

Hi Team,

Can you please help me with below issue.

We have created robots.txt file to deny access to crawlers on some paths but we are facing below issue.

 Steps we followed:

   1. Created robots.txt file in DAM path(/content/dam/folder)

   2. Added relevant content as per business rules

   3. Allowed robots.txt path by adding rules in filters& added rewrite rules in dispatcher and restarted apache server

   4. When we hit url(https:host:port/robots.txt) it's downloaded instead of opening.

Thanks,

Swamulu

[Raja-kp]

Can you tell me what's the issue here ?

[swamulu143]

Hi Raja,

Robots.txt file is downloading instead of opening file when we hit Url.

[Kishore_Kumar_ · Accepted solution]

Hi @swamulu143 ,

It might be due to content disposition, Can you please try adding a path in Apache Sling Content Disposition Filter as mentioned in below link.

https://experienceleaguecommunities.adobe.com/t5/adobe-experience-manager/how-to-allow-a-robots-txt-... 

[arunpatidar]

Hi,

Can you check if the response header, if it has this header then you can remove this header from httpd server for this file

Content-Disposition: attachment; filename="robots.txt"