Your achievements

Level 1

0% to

Level 2

Tip /
Sign in

Sign in to Community

to gain points, level up, and earn exciting badges like the new
BedrockMission!

Learn More

View all

Sign in to view all badges

Robots.txt in not working in Publish instance

Avatar

Avatar
Validate 1
Level 2
lovepreetk75597
Level 2

Likes

3 likes

Total Posts

24 posts

Correct Reply

0 solutions
Top badges earned
Validate 1
Ignite 5
Ignite 3
Ignite 1
Give Back 3
View profile

Avatar
Validate 1
Level 2
lovepreetk75597
Level 2

Likes

3 likes

Total Posts

24 posts

Correct Reply

0 solutions
Top badges earned
Validate 1
Ignite 5
Ignite 3
Ignite 1
Give Back 3
View profile
lovepreetk75597
Level 2

16-01-2020

Hi,

 

We have one requirement where we need to add robots.txt file into AEM to exclude our site from google search engine.. We have added the file on author and its working there but not working for publish instance.I have taken same package from author and import to publish instance but on publish we are getting 404 while trying to hit robots.txt directly through URL. Any help will be appreciated. Thanks in advance.

 

Regards,

Lovepreet 

Replies

Avatar

Avatar
Coach
MVP
Arun_Patidar
MVP

Likes

1,348 likes

Total Posts

3,226 posts

Correct Reply

916 solutions
Top badges earned
Coach
Contributor 2
Ignite 10
Give Back 700
Boost 1000
View profile

Avatar
Coach
MVP
Arun_Patidar
MVP

Likes

1,348 likes

Total Posts

3,226 posts

Correct Reply

916 solutions
Top badges earned
Coach
Contributor 2
Ignite 10
Give Back 700
Boost 1000
View profile
Arun_Patidar
MVP

16-01-2020

Are you accessing this vai dispatcher? Make sure .txt extension is enabled.

Avatar

Avatar
Coach
Employee
jbrar
Employee

Likes

387 likes

Total Posts

869 posts

Correct Reply

283 solutions
Top badges earned
Coach
Establish
Give Back 50
Give Back 5
Give Back 3
View profile

Avatar
Coach
Employee
jbrar
Employee

Likes

387 likes

Total Posts

869 posts

Correct Reply

283 solutions
Top badges earned
Coach
Establish
Give Back 50
Give Back 5
Give Back 3
View profile
jbrar
Employee

16-01-2020

In addition to what Arun mentioned, If you are getting 404 for the robot.txt file when directly accessing it via publisher(Bypassing dispatcher), make sure the anonymous user has access to the location of robot.txt file.

Avatar

Avatar
Ignite 1
Level 1
abhishekk962745
Level 1

Like

1 like

Total Posts

6 posts

Correct Reply

0 solutions
Top badges earned
Ignite 1
Validate 1
Give Back
Boost 1
View profile

Avatar
Ignite 1
Level 1
abhishekk962745
Level 1

Like

1 like

Total Posts

6 posts

Correct Reply

0 solutions
Top badges earned
Ignite 1
Validate 1
Give Back
Boost 1
View profile
abhishekk962745
Level 1

16-01-2020

If you are on  6.4 and higher, make sure you add your domain (www.xyz.com) into your vhosts.any file under /etc/httpd/conf.dispatcher.d/vhosts

Avatar

Avatar
Give Back
Level 2
amogh-personal
Level 2

Likes

11 likes

Total Posts

10 posts

Correct Reply

5 solutions
Top badges earned
Give Back
Boost 5
Boost 3
Boost 10
Boost 1
View profile

Avatar
Give Back
Level 2
amogh-personal
Level 2

Likes

11 likes

Total Posts

10 posts

Correct Reply

5 solutions
Top badges earned
Give Back
Boost 5
Boost 3
Boost 10
Boost 1
View profile
amogh-personal
Level 2

25-01-2020

Where that file is hosted? If it is under /content/you-project then try clearing your domain cache or if it is hosted under any specific restricted path then check the anonymous access. 

Thanks,
Amogh