Robots.txt in AEM codebase

[cquser1]

Hi All,

Have seen robots.txt in AEM codebase which has entries like User-agent: *
Disallow: /

Can someone let me know what exactly is the use of this file and what do these above properties in it signify.

Any thoughts on this will be helpful.

[Amit_Kumar]

Hi,

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.

Red more here: https://moz.com/learn/seo/robotstxt

Regards,

Amit

[Jitendra_S_Toma · Accepted solution]

Hi CQUser1,

The purpose of the Robots.txt is to provide details about the application to the search engine. One of the thing is to instruct web crawlers that a particular path is not index-able (Disallow: /).

i hope this helps.

cquser1 wrote...

Hi All,

Have seen robots.txt in AEM codebase which has entries like User-agent: *
Disallow: /

Can someone let me know what exactly is the use of this file and what do these above properties in it signify.

Any thoughts on this will be helpful.

[cquser1]

Hi Amit/Jitendra,

Thank you for your replies.