Robots.txt changes that silently deindex whole sections

A robots.txt file controls which URLs a crawler may fetch. It does not control what appears in search results, and that gap is where the damage happens. A developer cleaning up a staging rule adds one line, Disallow: /blog, and it ships with Tuesday’s deploy. The site works. The pages render. The tests pass. Googlebot stops fetching the blog within hours, and nobody finds out until the traffic chart bends a week later.

Why this file is the one nobody watches

Robots.txt is the only file on your site whose entire audience is a crawler. Humans never open it. Uptime checks never read it. It is usually generated, often by a plugin or a build step, and it sits outside every deploy checklist because nothing a human does in a browser ever touches it.

Recovery is slow in a way people underestimate. After you fix the line, Googlebot has to notice the new file, re-crawl the affected URLs, and re-process them. For a section that was quietly disallowed for 48 hours, expect one to three weeks before the charts look normal again.

Disallow is not noindex, and confusing them makes things worse

This is the single most common misunderstanding in this area, and acting on it usually causes the problem it was meant to fix.

• Disallow says: do not fetch this URL.
• noindex says: do not put this URL in the index.

A page that is disallowed can still appear in search results. Google knows the URL exists because other pages link to it, but it cannot fetch the page, so it lists the bare URL with no title or description. That is why disallowing a page to get it out of Google often leaves an uglier result than before.

Worse, the two directives interact badly. If you add both, Google cannot see the noindex, because seeing it would require fetching the page that robots.txt just told it not to fetch. The correct order is: allow the crawl, serve the noindex, wait for it to be processed, and only then disallow if you still want to.

How Googlebot actually resolves a robots.txt

The file’s text is the wrong thing to reason about. What matters is the answer for one specific URL and one specific crawler. Three rules produce that answer.

One group wins, and the rest are ignored. A crawler picks the single most specific group whose user-agent line names it, and obeys only that group. If your file has a User-agent: * group and a User-agent: Googlebot group, then Googlebot reads the Googlebot group and nothing else. Every rule you carefully added to the wildcard group is invisible to it. This is the trap that catches people who add a rule “for everyone” and cannot work out why Google ignored it.

The longest matching path wins. Within the chosen group, rules are not read top to bottom. Google compares every rule that matches the URL and takes the one with the longest path pattern:

User-agent: *
Disallow: /blog
Allow: /blog/public

For /blog/public/post, both rules match. Allow: /blog/public is longer, so the URL is allowed. For /blog/private/post, only the Disallow matches, so it is blocked.

Allow wins a tie. When two matching rules are the same length, Google takes the least restrictive one, which means Allow beats Disallow.

Two characters change matching. * matches any run of characters, and $ anchors to the end of the URL. So Disallow: /*.pdf$ blocks PDFs and nothing else, while Disallow: /*.pdf also blocks /report.pdf?download=1.

The failure modes, in the order they actually happen

A prefix that is wider than intended. Disallow: /b looks like it blocks a folder. It blocks /blog, /billing, and /bulk-orders too, because robots.txt matches path prefixes, not directories.

A rule added to the wrong group. Covered above, and worth checking first whenever a rule seems to have had no effect.

A generated file that regressed. A CMS update, a plugin, or a framework upgrade rewrites robots.txt and reverts your rules. Nothing in your repository changed, so a code review would never have caught it.

The environment variable that was wrong at build time. Static site generators commonly emit Disallow: / for non-production builds. Get the environment flag wrong once and you ship the staging robots.txt to production.

A 5xx instead of a 404. These are not equivalent. A missing robots.txt means no restrictions, and crawling continues normally. A server error means Google cannot determine what is allowed, so it backs off crawling the site until the file is readable. A robots.txt that throws a 500 during an incident quietly makes the incident worse.

A CDN serving a different file than the origin. Your origin is correct, the edge is not, and you cannot see it from the server.

What to check after a deploy

1. Fetch the live robots.txt from the public internet, not from the origin, so you see what a crawler sees through your CDN.
2. Confirm the status is 200 or 404. Anything else, especially a 5xx, is the first thing to fix.
3. For each URL that earns traffic, resolve the effective permission for Googlebot specifically, not just for *.
4. Compare that answer to the last known answer. The question is never “did the file change”, it is “did the answer for this URL flip from allowed to disallowed”.
5. Check that no URL in your sitemap is disallowed. A sitemap that lists blocked URLs sends two contradictory instructions.

Google retired its interactive robots.txt Tester in December 2023. The replacement report in Search Console shows the fetched file and its status, but only for verified properties, and it cannot test an arbitrary URL against the rules. If you used to paste a path into the old tester, that workflow no longer exists inside Google.

Watch the answer, not the file

Text churn is not a signal. Comments get reordered, a sitemap line gets added, a generator rewrites the whitespace, and none of it changes what any crawler is allowed to do. Alerting on the diff means alerting on noise, and a monitor you learn to ignore is worse than no monitor.

The event worth waking up for is narrow: an effective permission for a URL you care about flipped from allowed to disallowed, for a crawler that matters. Tuesday 14:31, a deploy lands. At 14:45 the check resolves /pricing against the new file, finds it disallowed for Googlebot where yesterday it was allowed, re-checks 60 seconds later to rule out a transient, and only then sends the alert with the exact before and after.

stillindexed.com does that as often as every 15 minutes, for * and Googlebot, against every URL you monitor. Test any robots.txt free, no account needed, or see pricing.