Blocked by robots.txt: find the rule and decide whether to unblock

Last updated: 2026-10-30

In short

Blocked by robots.txt means a rule in your robots.txt file stops Googlebot from crawling the URL, so Google cannot index its content. If the block is intentional, leave it. If the page should be in Google, remove or narrow the rule; if you want it out of Google, use noindex instead, because robots.txt does not reliably prevent indexing.

Exact label in Search Console: Blocked by robots.txt

What it means

Google's Page indexing help says the page was blocked by your site's robots.txt file, and warns that this does not guarantee the page will not be indexed some other way: if Google can find information about the page without loading it, there is a very small chance it still gets indexed. That case is reported separately as Indexed, though blocked by robots.txt. Google's help article titles this reason URL blocked by robots.txt; the report row reads Blocked by robots.txt.

Google describes robots.txt as a tool for managing crawler traffic, not a way to keep a page out of Google. Googlebot obeys it, but a blocked page can still appear in results without a description if other pages link to it. A robots.txt file applies only to its own protocol and host, and Google generally caches it for up to 24 hours, so changes are not picked up instantly.

Is it a problem?

Not if you meant it. Admin areas, cart and checkout, internal search and endless filter combinations are often blocked on purpose, and Google says URLs blocked by robots.txt are probably intentionally blocked. It is a problem when important pages or whole sections are caught by an overly broad rule, a staging robots.txt shipped to production or a CMS setting.

Common causes

Intentional blocks

Private, utility or crawl-trap URLs you chose to keep crawlers away from.

Overly broad patterns

A Disallow on a short path prefix, or a wildcard pattern, that matches far more URLs than intended.

Staging or launch leftovers

A blanket Disallow for all user agents copied over from a development site.

CMS or plugin settings

Platform settings that write robots.txt rules for you, sometimes covering a whole section.

A different file than you think

Each protocol and host has its own robots.txt. A subdomain or the HTTP version can carry different rules from the file you edited.

How to fix it

  1. Find the rule. Inspect an example URL: Crawl allowed? shows whether robots.txt blocks it. The robots.txt report in Search Console shows which robots.txt files Google found for your hosts, when each was last fetched and any parsing issues.
  2. Decide section by section. Keep blocks for URLs that should not be crawled. For pages that belong in Google, remove the matching Disallow line or add a more specific Allow rule.
  3. Use noindex to keep pages out of Google. If the goal is to keep a page out of search results, remove the robots.txt block and add noindex. Google cannot see a noindex rule on a page it is not allowed to crawl.
  4. Request a robots.txt recrawl after big changes. Google recrawls robots.txt often, but after unblocking important URLs you can request a recrawl of the file in the robots.txt report. Google notes this does not guarantee an immediate recrawl of the unblocked pages.
  5. Test and validate. Run Test live URL on unblocked pages, then click Validate fix if you unblocked a whole group of URLs.

Check it with GSC Magic

gsc_inspect_urls_batch reports the robots.txt state for each URL, so your assistant can confirm whether important pages are blocked and which directories a rule covers. report_index_health applies the same check to every URL in a sitemap, where nothing should be blocked. In the GSC Magic dashboard, the robots.txt and sitemap quick audit for each site shows the live robots.txt next to your sitemaps.

gsc_inspect_urls_batchreport_index_healthgsc_inspect_url

Prompts to copy

Inspect these URLs and list every one where crawling is disallowed by robots.txt. Group them by directory so I can see which rule is likely responsible.

Run an index health check on my sitemap and tell me whether any submitted URL is blocked by robots.txt.

Here is my robots.txt file. Tell me which of these 50 URLs each Disallow rule matches, and flag rules that block URLs listed in my sitemap.

robots.txt tester →

Guides on the blog

Indexed, though blocked by robots.txt

This warning means Google indexed the URL even though robots.txt blocks crawling it, usually because other pages link to it. Google cannot read the page, so any snippet is very limited. If you want the page out of Google, remove the block and add noindex; if you want it in Google, remove the block.

 Excluded by 'noindex' tag

This status means Google crawled the page, found a noindex rule in a robots meta tag or an X-Robots-Tag HTTP header, and therefore did not index it. If you do not want the page in Google, this is working. If you do, remove the noindex at its source, confirm with Test live URL and request indexing.

 Discovered - currently not indexed

Discovered - currently not indexed means Google found the URL, through a link or a sitemap, but has not crawled it yet. Google says it typically postponed the crawl because crawling was expected to overload the site. Fix it by cutting low-value URLs, improving server response and linking to the pages that matter.

Frequently asked questions

Does robots.txt keep a page out of Google?

No. Google says robots.txt is not a mechanism for keeping a web page out of Google. A blocked URL can still be indexed from links, without its content. Use noindex or password protection instead.

Can I put noindex in robots.txt?

No. Google says a noindex rule in robots.txt is not supported. Use a robots meta tag or the X-Robots-Tag HTTP header on the page itself.

How quickly does Google see a robots.txt change?

Google generally caches robots.txt for up to 24 hours. After an important change you can request a recrawl of the file in the robots.txt report.

What if robots.txt itself returns an error?

Google's robots.txt specification says that if the file returns a server error, Google stops crawling the site for the first 12 hours, then uses the last good copy for up to 30 days while retrying. A 4xx response other than 429 is treated as if no robots.txt existed.

Sources