Kick Butt Tip #23 – Block Googlebot from Unwanted Pages

If you want to block Google from accessing certain pages you do this in a robots.txt file. This is not news to many webmasters, but what you may not realize is you need to specifically block Google instead of relying on a global bot block.

For example, this is what many use in their robots.txt to block bots from accessing the /block subdirectory (this is a hypothetical subdirectory)

User-agent: *
Disallow: /block/

This is what need to ALSO include in your robots.txt file to make sure Google is blocked:

User-agent: Googlebot
Disallow: /block/

Blocking Google can be useful to prevent unwanted pages from appearing in Google's public search index, and to prevent Google from knowing about certain links or pages.

This entry was posted in Tips. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>