stillindexedStart monitoringOpen the app

Search Console Page indexing: Why pages aren't indexed

Blocked by robots.txt in Search Console: what to do

A robots.txt rule stops Googlebot fetching the URL. When that is intended, when it is not, how to find the rule and how long unblocking takes.

A rule in your robots.txt stops Googlebot fetching this URL, so Google cannot read the page. If the block is deliberate, nothing needs fixing. If the page should rank, find the Disallow rule that matches it, remove or narrow it, and confirm Googlebot is allowed before you ask Google to validate.

What "Blocked by robots.txt" means

Google could not crawl the URL because a rule in your site's robots.txt disallows it. Google's help page calls this status "URL blocked by robots.txt"; the report and its CSV export label it "Blocked by robots.txt". A block stops crawling, not indexing: Google warns the URL can still be indexed without its content if other pages link to it, which the report lists separately as "Indexed, though blocked by robots.txt".

The report shows "Blocked by robots.txt". Google's help page for the report calls the same status "URL blocked by robots.txt".

Is it a problem?

Depends on the URL

Often fine. Google says URLs blocked by robots.txt "are probably intentionally blocked": admin paths, internal search, cart and filter URLs. It is a problem when the list contains pages you want in Search, or when the count jumps after a deploy.

Causes, most likely first

  1. A Disallow rule broader than intended. Rules match on prefix, so "Disallow: /p" blocks /products and /pricing as well as /p/.
  2. A staging or pre-launch robots.txt with "Disallow: /" deployed to production, or never removed after launch.
  3. A CMS, plugin or platform setting rewriting robots.txt, such as a "discourage search engines" option or an SEO plugin's file editor.
  4. A wildcard pattern (* or $) that matches more URLs than its author tested.
  5. A Googlebot group that differs from the * group. Google follows only the most specific group that matches its user agent and ignores the others, so rules written for * stop applying to Googlebot once a Googlebot group exists.

How to confirm it

  1. Run the robots.txt tester with the URL. It resolves which rule wins for Googlebot and for the * group separately, from the live file.
  2. In URL Inspection, read "Crawl allowed?" on the indexed result and on a live test.
  3. On a domain-level property, open Property settings, robots.txt to see the version of the file Google last fetched, when, and any fetch errors.
  4. Open /robots.txt on the exact host in a browser. Each host, protocol and port has its own file, so www and non-www can differ.

Robots.txt tester

How to fix it

  1. Find the matching Disallow line. If the URL should be crawled, delete the rule or narrow it with a more specific path or an Allow line.
  2. If you meant to keep the page out of Search rather than out of the crawl, remove the robots.txt block and add noindex instead. Google can only see a noindex on a page it is allowed to fetch.
  3. Deploy the file, re-run the robots.txt tester, and on a domain-level property use Request a recrawl in the robots.txt report.
  4. Request indexing for the most important URLs in URL Inspection, then click Validate fix.

How long recovery takes

Google "generally caches the contents of robots.txt file for up to 24 hours", longer if it cannot refresh the file, so a corrected rule can take a day to be read. Blocked URLs "will be recrawled when the block is removed", and recrawling can take "anywhere from a few days to a few weeks".

After a fix, Validate fix on the issue asks Google to recheck the affected URLs. Google says validation "typically takes up to about two weeks, but in some cases can take much longer". Without it, the count still updates as Google recrawls, and an issue leaves the table once 90 days pass with no instance.

Would monitoring have caught it?

A robots.txt change that newly blocks a monitored URL for Googlebot, or for every crawler, is a critical change. The file is evaluated against each monitored path, so an edit that leaves your URLs allowed does not alert.

stillindexed reads robots.txt once a day for each site and evaluates it against every monitored path. A new block is recorded on the first reading that shows it and alerts when the next daily reading confirms it.

See what monitoring covers

robots.txt blocking pages that should be crawled, platform by platform

Further reading

Sources

Every claim above about how Google behaves is from Google's own documentation, read on 2026-10-06. If Google has changed a page since, Google's page wins and we would like to know.