Issue with Robots.txt

lovepreetk75597

19-02-2020

Hi,

 

I have added robots.txt to exclude our site from google search. I have excluded whole site by mentioning below content in robots.txt

 

user-agent: *

Disallow: /

 

But still some of the pages are visible in google search engine.

Any help would be appreciated.

 

Regards,

Lovepreet

Accepted Solutions (1)

Accepted Solutions (1)

BrianKasingli

MVP

20-02-2020

@lovepreetk75597 ,

Your configuration of the robots.txt file looks correct. Also, ensure that the robots.txt file can be accessed at the root level of your website.

e.g.: https://example.com/robots.txt

 

User-agent: *
Disallow: /

 

 One more thing is, you should request Google to recrawl your website. Check out this link, provided from Google's own knowledge centre, for instructions on how to ask Google to quickly crawl and index a new robots.txt https://support.google.com/webmasters/answer/6078399?hl=en

Answers (1)

Answers (1)

Arun_Patidar

MVP

19-02-2020

Blocked by robots.txt: This page was blocked to Googlebot with a robots.txt file. You can verify this using the robots.txt tester. Note that this does not mean that the page won't be indexed through some other means. If Google can find other information about this page without loading it, the page could still be indexed (though this is less common). To ensure that a page is not indexed by Google, remove the robots.txt block and use a 'noindex' directive.

 

https://support.google.com/webmasters/thread/6295192?hl=en