Issue with Robots.txt

[lovepreetk75597]

Hi,

I have added robots.txt to exclude our site from google search. I have excluded whole site by mentioning below content in robots.txt

user-agent: *

Disallow: /

But still some of the pages are visible in google search engine.

Any help would be appreciated.

Regards,

Lovepreet

[arunpatidar]

Blocked by robots.txt: This page was blocked to Googlebot with a robots.txt file. You can verify this using the robots.txt tester. Note that this does not mean that the page won't be indexed through some other means. If Google can find other information about this page without loading it, the page could still be indexed (though this is less common). To ensure that a page is not indexed by Google, remove the robots.txt block and use a 'noindex' directive.

https://support.google.com/webmasters/thread/6295192?hl=en

[BrianKasingli · Accepted solution]

@lovepreetk75597 ,

Your configuration of the robots.txt file looks correct. Also, ensure that the robots.txt file can be accessed at the root level of your website.

e.g.: https://example.com/robots.txt

User-agent: *
Disallow: /

 One more thing is, you should request Google to recrawl your website. Check out this link, provided from Google's own knowledge centre, for instructions on how to ask Google to quickly crawl and index a new robots.txt https://support.google.com/webmasters/answer/6078399?hl=en