robots.txt in AEM 6.3 - definitive AEM documentation on where it goes? | Community
Skip to main content
nesfel_aem
nesfel_aem
Level 3
January 2, 2018
Solved

robots.txt in AEM 6.3 - definitive AEM documentation on where it goes?

  • January 2, 2018
  • 6 replies
  • 10514 views

For non-AEM websites, there's usually a clearly defined web root folder where you can place files like robots.txt.  In AEM (v6.3), it's a bit more complicated with the Author, Publish, and Dispatcher instances.  Adding a robots.txt should be simple, but I'm not sure how to do it in AEM, and I can't find definitive documentation.  We don't need info about what the robots.txt file should contain--we just need the steps for getting it on our new AEM site.

Do we simply send our robots.txt file to a server administrator, who then adds the file on the Dispatcher servers?  Or should we use CRX to create the robots.txt file in Author, replicate to Publish, and add a redirect rule for the Dispatcher?  We have one Author instance, two Publish Instances, and two Dispatchers.  I'm hoping this is officially documented, and someone is kind enough to share the link.

Thanks for any help you can offer.

-Nestor

    This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.
    Best answer by thilaks18944609

    I would approach this in a different way.
    We need robot.txt to be authorable. It keeps changing time to time. adding it to /crx/de needs the code deployment to change that file.

    I think it is good for authors to have the ability to change the robot.txt file.
    you can add the robot.txt file under dam (some path like this /content/dam/yoursite/robots/). This lets authors edit the file in place(6.2 and above) and publish it.

    You can use one of the following below to redirect the robot.txt to point to the file:

    Sling mappings:

    sling:internalRedirect="/content/dam/yoursite/robots/robots.txt"

    sling:match="robots.txt"/

    Dispatcher rewrite rules:

    RewriteCond %{REQUEST_URI} ^/robots.txt

    RewriteRule ^/(.*) /content/dam/yoursite/robots/robots.txt [PT,NC,L]

    6 replies

    smacdonald2008
    smacdonald2008
    Level 10
    January 2, 2018

    Here is a community blog that may help: Adobe CQ/Adobe AEM: How to implement robots.txt / sitemap.xml / crossdomain.xml in Adobe CQ / AEM

    Ratna_Kumar
    Ratna_Kumar
    Level 10
    January 2, 2018

    Hi,

    Since there is no Adobe documentation on robots.txt, you can see the link which Scott pointed out: Adobe CQ/Adobe AEM: How to implement robots.txt / sitemap.xml / crossdomain.xml in Adobe CQ / AEM

    Thanks,

    Ratna.

    thilaks18944609
    thilaks18944609Accepted solution
    Level 2
    January 2, 2018

    I would approach this in a different way.
    We need robot.txt to be authorable. It keeps changing time to time. adding it to /crx/de needs the code deployment to change that file.

    I think it is good for authors to have the ability to change the robot.txt file.
    you can add the robot.txt file under dam (some path like this /content/dam/yoursite/robots/). This lets authors edit the file in place(6.2 and above) and publish it.

    You can use one of the following below to redirect the robot.txt to point to the file:

    Sling mappings:

    sling:internalRedirect="/content/dam/yoursite/robots/robots.txt"

    sling:match="robots.txt"/

    Dispatcher rewrite rules:

    RewriteCond %{REQUEST_URI} ^/robots.txt

    RewriteRule ^/(.*) /content/dam/yoursite/robots/robots.txt [PT,NC,L]

    nesfel_aem
    nesfel_aemAuthor
    Level 3
    January 2, 2018

    Thanks all three of you for your helpful replies.  While technically I was looking for official documentation, it's good to know that there are several options for implementing a solution.  shankarappa​, your approach seems like a very flexible solution that will best meet our needs long-term, so chose yours as the correct answer--much appreciated.  Again, thanks smacdonald2008​ and Ratna Kumar​ as well.

    U
    ushab71743342
    April 25, 2018

    Hi @thilaks18944609,

    I have to redirect /content/ifn/greatlakes/robots.txt to /robots.txt page as internal redirection of page properties is not working.

    AEM (v6.2/SP1),

    URL:http://greatlakes.pennmutual.com/content/ifn/greatlakes/robots.txt should be internal redirected to http://greatlakes.pennmutual.com/robots.txt

    I used following values under /etc/map/http folder :

    Created a node

    sling:match as "robots.txt/"

    sling:internalRedirect as "/content/ifn/greatlakes/robots.txt"

    my required redirection is not working .

    Please let me know correct configuration for the same .

    joerghoh
    Adobe Employee
    joerghohAdobe Employee
    Adobe Employee
    April 26, 2018

    There is no "officlal" documentation for this, because there is no single unified content structure for all use cases where AEM is used. Also there are many usecases and requirements how such a file is created or maintained.

    One approach is to store the robots.txt in DAM and reference it from there. Another one is to create it dynamically with a servlet based on a data stored in the content itself.

    Jörg

      Terms & ConditionsAccessibility statement

      Loading footer...