Issue with Robots.txt | Community
Skip to main content
lovepreetk75597
lovepreetk75597
Level 2
February 19, 2020
Solved

Issue with Robots.txt

  • February 19, 2020
  • 2 replies
  • 1924 views

Hi,

 

I have added robots.txt to exclude our site from google search. I have excluded whole site by mentioning below content in robots.txt

 

user-agent: *

Disallow: /

 

But still some of the pages are visible in google search engine.

Any help would be appreciated.

 

Regards,

Lovepreet

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.
Best answer by BrianKasingli

@lovepreetk75597 ,

Your configuration of the robots.txt file looks correct. Also, ensure that the robots.txt file can be accessed at the root level of your website.

e.g.: https://example.com/robots.txt

 

User-agent: * Disallow: /

 

 One more thing is, you should request Google to recrawl your website. Check out this link, provided from Google's own knowledge centre, for instructions on how to ask Google to quickly crawl and index a new robots.txt https://support.google.com/webmasters/answer/6078399?hl=en

2 replies

arunpatidar
Community Advisor
arunpatidarCommunity Advisor
Community Advisor
February 19, 2020

Blocked by robots.txt: This page was blocked to Googlebot with a robots.txt file. You can verify this using the robots.txt tester. Note that this does not mean that the page won't be indexed through some other means. If Google can find other information about this page without loading it, the page could still be indexed (though this is less common). To ensure that a page is not indexed by Google, remove the robots.txt block and use a 'noindex' directive.

 

https://support.google.com/webmasters/thread/6295192?hl=en

 

Arun Patidar
BrianKasingli
Community Advisor and Adobe Champion
BrianKasingliCommunity Advisor and Adobe ChampionAccepted solution
Community Advisor and Adobe Champion
February 20, 2020

@lovepreetk75597 ,

Your configuration of the robots.txt file looks correct. Also, ensure that the robots.txt file can be accessed at the root level of your website.

e.g.: https://example.com/robots.txt

 

User-agent: * Disallow: /

 

 One more thing is, you should request Google to recrawl your website. Check out this link, provided from Google's own knowledge centre, for instructions on how to ask Google to quickly crawl and index a new robots.txt https://support.google.com/webmasters/answer/6078399?hl=en

Terms & ConditionsAccessibility statement

Loading footer...