This article explains how to block access to content on your site.
Some of the content that you publish may not be relevant to appear on Google News. You can restrict Google’s access to certain content by blocking access to Google's robot crawlers, Googlebot and Googlebot-News.
Create a robots.txt file
Use a robots.txt file to get a high level of control over which parts of your site may appear in Google Search and Google News. Learn more about robots.txt files.
You can block access in the following ways:
-
To prevent your site from appearing in Google News, block access to Googlebot-News using a robots.txt file.
-
To prevent your site from appearing in Google News and Google Search, block access to Googlebot using a robots.txt file.
You need to give our crawler access to your robots.txt file so that we can see if you've specified certain sections of your site that you don't want crawled.
Create a meta tag
You can add meta tags to an HTML page. The meta tags tell search engines which limits apply when showing pages in search results. Learn how to block search indexing with meta tags.
Here are some common meta tags that you can add to your HTML pages:
-
To prevent specific articles on your site from appearing in Google News, block access to Googlebot-News using the following meta tag:
<meta name="Googlebot-News" content="noindex, nofollow">. -
To prevent specific articles on your site from appearing in Google News and Google Search, block access to Googlebot using the following meta tag:
<meta name="googlebot" content="noindex, nofollow">. -
To prevent specific articles on your site from being indexed by all robots, use the following meta tag:
<meta name="robots" content="noindex, nofollow">. -
To prevent robots from crawling images on a specific article, use the following meta tag:
<meta name="robots" content="noimageindex">. -
To inform us that an article should be removed from the Google index at a certain time, use the following meta tag:
<meta name="googlebot" content="unavailable_after: 25-Aug-2011 15:00:00 EST">. -
Specify the time and date in RFC 850 format. This meta tag is treated as a removal request. It takes about a day after the removal date passes for the page to disappear from the search results. However, for the tag to function properly, it must be included with your article when it’s first crawled.
-
There are other options for limiting the content shown in a search result. Find out more in the developer documentation.
HTTP header specifications
You can also provide instructions to robots in the HTTP response header. To learn more, read about HTTP header specifications.
Important: Google follows the most restrictive interpretation of your bot's choice.