Making sure your site is fully crawlable can help you earn more revenue from your content. If the content crawler can’t access your content, refer to the following list of crawler issues to help you fix the problem.
Access issues
Access issues| Possible reason |
What to check |
How to fix |
|---|---|---|
| Your site is behind a login.
It's possible that the ads crawler is being redirected to a login page, which means it can't crawl your content. |
Do visitors need login details to access your content? | Set up a crawler login to use web content behind login for content mapping in apps. |
| Your content is behind a restricted network or IP range. | Have you put restrictions in place that limit the geographies or IP ranges that can access your content? | Consider removing these restrictions or making your content publicly accessible, so that the ads crawler can crawl your URLs. |
| You've blocked the ads crawler in your robots.txt file. | Is the ads crawler disallowed in your robots.txt file? | Give access to the ads crawler in your robots.txt file. |
Site issues
| Possible reason |
What to check |
How to fix |
|---|---|---|
|
Your site can't be found. If the URL sent to Google points to a page that doesn't exist (or no longer exists) on a site, or results in a 404 (Not Found) error, the ads crawler won't successfully crawl your content. |
Is your site up and running? Can visitors access your site on the web? | Make sure your site is published and live on the web. |
|
Your site has freshly published content. When you publish a new page, you may make ad requests before Google's crawlers have had a chance to crawl your content. Examples of sites that post lots of new content include sites with user-generated content, news articles, large product inventories or weather sites. |
Have you recently published new content? |
Allow more time for the ads crawler to crawl your new content. Usually after the ad request is made on a new URL, the content will get crawled within a few minutes. However, during these initial few minutes, because your content has not yet been crawled, you may experience low ad volume. |
|
Your site uses multiple redirects. If your site uses redirects, there's a risk that the ads crawler might have issues following through them. For example, if you have multiple redirects and intermediate redirects fail, or if important parameters such as cookies get dropped during redirection, this could decrease the quality of crawling. |
Does your site use lots of redirects? Have you set up redirects on pages that you're showing ads? |
Consider minimizing the use of redirects on pages with ad code. Make sure all your redirects are implemented properly. |
|
You're using personalized page URLs. Some sites include extra parameters in their URLs that indicate the user who is logged in (e.g., a Session ID), or other information that may be unique to each visit. When this happens, the ads crawler may treat the URL as a new page, even if the content is the same. This could result in a few minutes lag time between the first ad request on the page and when the page gets crawled, as well as an increase in the crawler load on your servers. |
Does your site use URLs with extra parameters or dynamically generated URL paths? | Generally, if the content on a page doesn't change, consider removing the parameters from the URL and persisting that information another way. Having a simpler URL structure can help make your site more easily crawlable. |
|
You're using POST data. If your site sends POST data along with URLs (for example, passing form data via a POST request), it's possible that your site is rejecting requests that aren't accompanied by POST data. Note that since the ad crawler won't provide any POST data, this kind of setup can prevent the crawler from accessing your page. |
Does your site send POST data along with URLs? | If your page's content is determined by the data that the user inputs to a form, consider using a GET request. |
Hosting issues
Hosting issues| Possible reason |
What to check |
How to fix |
|---|---|---|
|
There's an issue with your site's name server. |
Is your name server set up correctly? Do you have any restrictions on where requests can come from? | Make sure the name server for your domain or subdomain is properly directing the ads crawler to your content. |
|
There's an issue with your site's server. Sometimes when the ads crawler tries to access site content, the site's server is unable to respond in time. This can happen because the server is down, slow or overloaded by requests. |
Is your site's server up and running? | We recommend ensuring your site is hosted on a reliable server or by a reliable service provider. |