Your achievements

Level 1

0% to

Level 2

Tip /
Sign in

Sign in to Community

to gain points, level up, and earn exciting badges like the new
Bedrock Mission!

Learn more

View all

Sign in to view all badges

SOLVED

How to allow a robots.txt file to be visible on browser when accessed by url

Avatar

Level 2

How to allow a robots.txt file to be visible on browser when accessed by url
We are trying this on AEM as a cloud and in dispatcher we have already pasted the following config in dispatcher_vhost.conf but it still doesnt seem to work.
<Directory "/mnt/var/www/html/content/dam/sitefolder/">

<FilesMatch "\.(txt)$">

Header set Content-Disposition "inline"

Header set Cache-Control "max-age=86400, public"

Require all granted

</FilesMatch>

</Directory>

1 Accepted Solution

Avatar

Correct answer by
Community Advisor

Hi @Ralph8,

By default, all the resources which has jcr:data property or jcr:content/jcr:data (which your robots.txt file will have in this case) will have an header with value as "attachment".

We need to make an entry (Exclude entry to not have the Content-disposition as attachment) in Apache Sling Content Disposition filter (in your publish instance) -> Clear cache, if desired and then try accessing the file. 

Vijayalakshmi_S_0-1636559848952.png

View solution in original post

2 Replies

Avatar

Community Advisor

@Ralph8 

1. Please add the below rule in the /filter section of the farm file:

##Add allow rule for robots.txt in dispatcher
/0100 { /type "allow"  /url "/robots.txt"}

 

2. Add the redirect rule in the vhost file or rewrite file:

RewriteRule ^/robots.txt$  https://<dns>/<content-path>/robots.txt   (your content path should be masked based on content masking rule or mapping rule in publisher)

 

Now - try to access the robots.txt with public domain.

 

 

Avatar

Level 2

@Raja-kp 
robots.txt file is getting downloaded its still the same.
I would need the file contents to be visible on browser on accessing..

Avatar

Correct answer by
Community Advisor

Hi @Ralph8,

By default, all the resources which has jcr:data property or jcr:content/jcr:data (which your robots.txt file will have in this case) will have an header with value as "attachment".

We need to make an entry (Exclude entry to not have the Content-disposition as attachment) in Apache Sling Content Disposition filter (in your publish instance) -> Clear cache, if desired and then try accessing the file. 

Vijayalakshmi_S_0-1636559848952.png

Avatar

Community Advisor

It means you are able to access the robots.txt with public domain but the file is getting downloaded - right?