Indexed, though blocked by robots.txt: unblock it or noindex it

Last updated: 2026-10-30

In short

This warning means Google indexed the URL even though robots.txt blocks crawling it, usually because other pages link to it. Google cannot read the page, so any snippet is very limited. If you want the page out of Google, remove the block and add noindex; if you want it in Google, remove the block.

Exact label in Search Console: Indexed, though blocked by robots.txt

What it means

Google lists this under warnings in the Improve page experience table of the Page indexing report: issues that do not prevent indexing but reduce Google's ability to understand your pages. Google says it always respects robots.txt, but that does not necessarily prevent indexing if someone links to the page. Google will not request and crawl the page, but it can still index it using information from the pages that link to it, and any snippet shown in results will probably be very limited.

Google's robots.txt guide says the same: a disallowed URL can still appear in results, potentially with the URL and the anchor text of links, but without a description. URL Inspection adds a warning that for a page blocked by robots.txt, Indexing allowed? always shows Yes, because Google cannot see any noindex directive on it.

Is it a problem?

It depends on what you wanted. If you blocked the URL to keep it out of Google, the block did not do that job and you need noindex instead. If the URL should be in Google with a proper snippet, the block is holding it back. Either way, robots.txt is the wrong tool for the result you are after.

Common causes

Using robots.txt to hide pages

Blocking pages in robots.txt in the hope of keeping them out of search, while they are linked from your own or other sites.

Broad rules over linked URLs

Disallow rules covering sections people link to, such as PDFs, parameter URLs or old directories.

Blocking on top of noindex

Adding a robots.txt block to a page that already has noindex, which stops Google from ever seeing the noindex.

Blocked URLs in sitemaps

Sitemaps that list URLs your robots.txt disallows. Google names this as a cause of error spikes in the report.

How to fix it

  1. Decide whether each page belongs in Google. Go through the example URLs and sort them into pages you want indexed and pages you want gone.
  2. To remove it: unblock and noindex. Remove the robots.txt rule for those URLs and add a robots meta noindex tag or X-Robots-Tag header. Once Google recrawls and sees noindex, it drops the page. For urgent cases, the Removals tool hides a URL for about six months while noindex takes effect.
  3. To keep it: unblock it. Remove or narrow the Disallow rule so Google can crawl the page and build a full snippet. Check in URL Inspection that Crawl allowed? now says Yes.
  4. Clean up links and sitemaps. Remove blocked URLs from your sitemaps and stop linking to URLs you do not want indexed.
  5. Request a robots.txt recrawl. After editing the file, request a recrawl in the robots.txt report so Google picks up the change sooner.

Check it with GSC Magic

gsc_inspect_urls_batch returns both the robots.txt state and the index verdict for each URL, so your assistant can list URLs that are indexed while disallowed and group them by the directory a rule covers. gsc_query_analytics shows whether those URLs actually get impressions, which helps you choose between unblocking and noindexing, and report_index_health finds disallowed URLs inside your sitemap. The GSC Magic dashboard has a robots.txt and sitemap quick audit that shows the live rules next to your sitemaps.

gsc_inspect_urls_batchgsc_query_analyticsgsc_inspect_urlreport_index_health

Prompts to copy

Inspect these URLs and list every one that is indexed even though robots.txt disallows crawling, grouped by directory.

For URLs that are indexed though blocked by robots.txt, show impressions and clicks over the last 3 months so I can decide which to unblock and which to noindex.

Run an index health check on my sitemap and list any submitted URL that robots.txt blocks.

robots.txt tester →

Guides on the blog

Blocked by robots.txt

Blocked by robots.txt means a rule in your robots.txt file stops Googlebot from crawling the URL, so Google cannot index its content. If the block is intentional, leave it. If the page should be in Google, remove or narrow the rule; if you want it out of Google, use noindex instead, because robots.txt does not reliably prevent indexing.

 Excluded by 'noindex' tag

This status means Google crawled the page, found a noindex rule in a robots meta tag or an X-Robots-Tag HTTP header, and therefore did not index it. If you do not want the page in Google, this is working. If you do, remove the noindex at its source, confirm with Test live URL and request indexing.

 Page indexed without content

This warning means the URL is in Google's index, but Google could not read the page content. Google names two possible reasons: the page may be cloaked to Google, or it may be in a format Google cannot index. Inspect the URL and check what Googlebot actually received.

Frequently asked questions

Why did Google index a page it may not crawl?

Because blocking crawling is not the same as blocking indexing. Google says it can index the URL using information from pages that link to it, without crawling the page itself.

Can I add noindex and keep the robots.txt block?

No. Google cannot see noindex on a page it is not allowed to crawl. Remove the block first, then add noindex.

Is this an error?

Google lists it as a warning, not as a reason a page is not indexed. The page is indexed, but with a very limited snippet and without Google reading its content.

How do I remove these URLs quickly?

Use the Removals tool in Search Console, which Google says hides a URL for about six months, and add noindex with crawling allowed so the removal becomes permanent.

Sources