Skip to main content
cquser1
cquser1
Level 7
May 3, 2017 4:30 AM UTC
Solved

Robots.txt in AEM codebase

Hi All,

Have seen robots.txt in AEM codebase which has entries like User-agent: *
Disallow: /

Can someone let me know what exactly is the use of this file and what do these above properties in it signify.

Any thoughts on this will be helpful.

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.
Jitendra_S_Toma
Jitendra_S_TomaAccepted solution
Level 10
May 4, 2017 8:23 AM UTC

Hi CQUser1,

The purpose of the Robots.txt is to provide details about the application to the search engine. One of the thing is to instruct web crawlers that a particular path is not index-able (Disallow: /).

i hope this helps.

cquser1 wrote...

Hi All,

Have seen robots.txt in AEM codebase which has entries like User-agent: *
Disallow: /

Can someone let me know what exactly is the use of this file and what do these above properties in it signify.

Any thoughts on this will be helpful.

 

View in thread

3 replies

Amit_Kumar
Amit_Kumar
Level 10
May 4, 2017 5:42 AM UTC

Hi,

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.

Red more here: https://moz.com/learn/seo/robotstxt

Regards,

Amit

Translate
Jitendra_S_Toma
Jitendra_S_TomaAccepted solution
Level 10
May 4, 2017 8:23 AM UTC

Hi CQUser1,

The purpose of the Robots.txt is to provide details about the application to the search engine. One of the thing is to instruct web crawlers that a particular path is not index-able (Disallow: /).

i hope this helps.

cquser1 wrote...

Hi All,

Have seen robots.txt in AEM codebase which has entries like User-agent: *
Disallow: /

Can someone let me know what exactly is the use of this file and what do these above properties in it signify.

Any thoughts on this will be helpful.

 

Translate
cquser1
cquser1Author
Level 7
May 4, 2017 9:54 AM UTC

Hi Amit/Jitendra,

Thank you for your replies.

Translate
Terms & ConditionsAccessibility statement

Loading footer...