Search Console Page indexing: Improve page experience

Indexed, though blocked by robots.txt: the fix

Google indexed a URL it is not allowed to crawl. Why robots.txt does not keep pages out of Search, and what to do whether you want the page or not.

Google indexed this URL without reading it: robots.txt blocks crawling, but links elsewhere told Google the page exists. To keep it out of Search, remove the robots.txt block and add noindex. To have it in Search, remove the block so Google can read the page.

What "Indexed, though blocked by robots.txt" means

Google: "The page was indexed despite being blocked by your website's robots.txt file." Google respects the block and does not crawl the page, but "we can still index it, using the information from the page that links to your blocked page." The result usually shows the URL with little or no description. The report lists this as a warning under Improve page experience, not under Why pages aren't indexed.

Is it a problem?

Usually a problem

Yes, in either direction. If you wanted the page hidden, robots.txt did not hide it. If you wanted it ranking, Google cannot read its content and any snippet will be very limited.

Causes, most likely first

robots.txt used to keep pages out of Search. Google says it is not the mechanism for that: a disallowed page can still be indexed if other pages link to it.
A staging, admin or internal section blocked in robots.txt but linked from somewhere public.
A Disallow rule broader than intended, catching pages you want indexed.
A noindex added to pages that robots.txt also blocks. Google cannot fetch the page, so it never sees the noindex.

How to confirm it

Run the robots.txt tester with the URL to find the rule that blocks Googlebot.
In URL Inspection, "Crawl allowed?" will read No. Google notes that while robots.txt blocks a page, "Indexing allowed?" always reads Yes, because Google cannot see a noindex.
Run the indexability checker to see whether the page carries a noindex that Google is currently unable to read.

Robots.txt tester Indexability checker

How to fix it

To keep the page out of Search: remove its robots.txt rule, add a noindex meta tag or X-Robots-Tag header, and leave the page crawlable until Google has recrawled it. For a whole staging host, password protection is the stronger option.
To get the page into Search properly: remove or narrow the Disallow rule so Googlebot can fetch it.
If it has to disappear from results while the noindex takes effect, a Removals request hides it temporarily.
Click Validate fix once the block is gone.

How long recovery takes

Google caches robots.txt "for up to 24 hours", so a change can take a day to be read. The page then has to be recrawled, which can take "anywhere from a few days to a few weeks". A Removals request is temporary: Google says a successful one "lasts only about six months".

After a fix, Validate fix on the issue asks Google to recheck the affected URLs. Google says validation "typically takes up to about two weeks, but in some cases can take much longer". Without it, the count still updates as Google recrawls, and an issue leaves the table once 90 days pass with no instance.

Would monitoring have caught it?

A robots.txt change that newly blocks a monitored URL for Googlebot, or for every crawler, is a critical change. A noindex appearing on a monitored URL is reported too, on the URL check cadence.

stillindexed reads robots.txt once a day for each site and evaluates it against every monitored path. A new block is recorded on the first reading that shows it and alerts when the next daily reading confirms it.

See what monitoring covers

robots.txt blocking pages that should be crawled, platform by platform

A staging or preview site indexed by Google, platform by platform

Sources

Every claim above about how Google behaves is from Google's own documentation, read on 2026-10-06. If Google has changed a page since, Google's page wins and we would like to know.