robots.txt in AEM 6.3 - definitive AEM documentation on where it goes?

[nesfel_aem]

For non-AEM websites, there's usually a clearly defined web root folder where you can place files like robots.txt.  In AEM (v6.3), it's a bit more complicated with the Author, Publish, and Dispatcher instances.  Adding a robots.txt should be simple, but I'm not sure how to do it in AEM, and I can't find definitive documentation.  We don't need info about what the robots.txt file should contain--we just need the steps for getting it on our new AEM site.

Do we simply send our robots.txt file to a server administrator, who then adds the file on the Dispatcher servers?  Or should we use CRX to create the robots.txt file in Author, replicate to Publish, and add a redirect rule for the Dispatcher?  We have one Author instance, two Publish Instances, and two Dispatchers.  I'm hoping this is officially documented, and someone is kind enough to share the link.

Thanks for any help you can offer.

-Nestor

[smacdonald2008]

Here is a community blog that may help: Adobe CQ/Adobe AEM: How to implement robots.txt / sitemap.xml / crossdomain.xml in Adobe CQ / AEM

[Ratna_Kumar]

Hi,

Since there is no Adobe documentation on robots.txt, you can see the link which Scott pointed out: Adobe CQ/Adobe AEM: How to implement robots.txt / sitemap.xml / crossdomain.xml in Adobe CQ / AEM

Thanks,

Ratna.

[thilaks18944609 · Accepted solution]

I would approach this in a different way.
We need robot.txt to be authorable. It keeps changing time to time. adding it to /crx/de needs the code deployment to change that file.

I think it is good for authors to have the ability to change the robot.txt file.
you can add the robot.txt file under dam (some path like this /content/dam/yoursite/robots/). This lets authors edit the file in place(6.2 and above) and publish it.

You can use one of the following below to redirect the robot.txt to point to the file:

Sling mappings:

sling:internalRedirect="/content/dam/yoursite/robots/robots.txt"

sling:match="robots.txt"/

Dispatcher rewrite rules:

RewriteCond %{REQUEST_URI} ^/robots.txt

RewriteRule ^/(.*) /content/dam/yoursite/robots/robots.txt [PT,NC,L]

[nesfel_aem]

Thanks all three of you for your helpful replies.  While technically I was looking for official documentation, it's good to know that there are several options for implementing a solution.  shankarappa​, your approach seems like a very flexible solution that will best meet our needs long-term, so chose yours as the correct answer--much appreciated.  Again, thanks smacdonald2008​ and Ratna Kumar​ as well.

[ushab71743342]

Hi thilaks18944609,

I have to redirect /content/ifn/greatlakes/robots.txt to /robots.txt page as internal redirection of page properties is not working.

AEM (v6.2/SP1),

URL:http://greatlakes.pennmutual.com/content/ifn/greatlakes/robots.txt should be internal redirected to http://greatlakes.pennmutual.com/robots.txt

I used following values under /etc/map/http folder :

Created a node

sling:match as "robots.txt/"

sling:internalRedirect as "/content/ifn/greatlakes/robots.txt"

my required redirection is not working .

Please let me know correct configuration for the same .

[Jörg_Hoh]

There is no "officlal" documentation for this, because there is no single unified content structure for all use cases where AEM is used. Also there are many usecases and requirements how such a file is created or maintained.

One approach is to store the robots.txt in DAM and reference it from there. Another one is to create it dynamically with a servlet based on a data stored in the content itself.

Jörg