Hi community,
We have set up an Edge Delivery site (https://github.com/hlxsites/aem-boilerplate-commerce) using the Cloud Manager portal:
https://main--saas-repo--myUserName.aem.live/
Currently, the site is hosted on a .aem.live or .aem.page domain, and the default robots.txt content is being displayed.
# AEM robots.txt FAQ
#
# Q: This looks like a default robots.txt, how can I provide my own?
# A: Add the contents of your robots.txt to your site configuration:
# https://labs.aem.live/tools/robots-edit/index.html
#
# Q: Why am I seeing this robots.txt instead of the one I
# configured?
# A: You are visiting from *.aem.page or *.aem.live - in order
# to prevent these sites from showing up in search engines and
# giving you a duplicate content penalty on your real site we
# exclude all robots
#
# Q: What do you mean with "real site"?
# A: If you add a custom domain to this site (e.g.
# example.com), then Franklin detects that you are ready for
# production and serves your own robots.txt - but only on
# example.com
#
# Q: This does not answer my questions at all. What can I do?
# A: Head to your Teams or Slack channel, or ask us on Discord:
# https://discord.gg/aem-live
User-agent: *
Disallow: /
After mapping the custom domain, the robots.txt now displays:
User-Agent: *
Allow: /
Sitemap: https://customdomain.com/sitemap.xml
However, we need to update the content of the robots.txt file.
We attempted to use the AEM tools for editing the robots.txt, but the tool indicates that the robots.txt file is not available at the root of the website:
https://labs.aem.live/tools/robots-edit/index.html
Additionally, we added a robots.txt file in the root of the main branch of the website, but it is still not functioning as expected.
Thank you in advance for your assistance.
