I have to create robots.txt file for my project.
We are using AEMaaCS. My questions are -
1. Where should I put the file? Should it be under /content/dam/project_name or somewhere else?
2. Should it be generated manually or dynamically using a servlet?
Are there any good and updated resources to refer? Currently, I am referring the below but these are a bit unclear and pretty old as well.
http://www.wemblog.com/2013/06/how-to-implement-robotstxt-sitemapxml.html
https://adobe-consulting-services.github.io/acs-aem-commons/features/robots-txt/index.html
Can someone please guide me in the right direction?
Solved! Go to Solution.
Views
Replies
Total Likes
In AEM, the robots.txt file should ideally be accessible at the root URL of your website, which is typically in the format https://www.example.com/robots.txt. However, when using AEM as a Cloud Service (AEMaaCS) and considering the content structure, the robots.txt file placed under /content/dam/project_name may not be directly accessible at the root URL.
To ensure the robots.txt file is accessible at the root URL you may need to configure the CDN rules to map the root URL to the location of the robots.txt file. This involves setting up specific rules in your CDN configuration to redirect requests for https://www.example.com/robots.txt to the actual location of the robots.txt file in AEM.
Hi @goyalkritika ,
Refer below blog for the location of the robots.txt file it will give you detailed insight about location
https://www.aemtutorial.info/2020/07/robotstxt-file-in-aem-websites.html
https://www.youtube.com/watch?v=3xy-z41Isws
Manual vs. dynamic generation:
@MayurSatav based on my project requirements, I'll go with manual creation.
So, I created the file under /content/dam/project_name with required permissions and published it.
Should robots.txt be not accessible as domain_name/robots.txt?
In this case, it will not be accessible like this as we will not be doing any masking.
Any insights?
In AEM, the robots.txt file should ideally be accessible at the root URL of your website, which is typically in the format https://www.example.com/robots.txt. However, when using AEM as a Cloud Service (AEMaaCS) and considering the content structure, the robots.txt file placed under /content/dam/project_name may not be directly accessible at the root URL.
To ensure the robots.txt file is accessible at the root URL you may need to configure the CDN rules to map the root URL to the location of the robots.txt file. This involves setting up specific rules in your CDN configuration to redirect requests for https://www.example.com/robots.txt to the actual location of the robots.txt file in AEM.
Views
Likes
Replies