stillindexedStart monitoringOpen the app

What is noindex, and why it silently kills traffic

noindex tells Google not to index a page. Here is how it works, why a stray noindex routinely sits live for 24 to 72 hours before anyone notices, and how to find and fix it before rankings slide.

A noindex directive is a single line that removes a page from Google’s index. It is the worst thing that can go wrong on a site you manage, for one reason: it does not throw an error, it does not change the look of the page, and no user ever reports it. The only thing that changes is whether Googlebot is allowed to keep the page in the index.

What noindex actually is

There are two ways to send it, and the first one is invisible in a normal page view:

  • Meta robots: a <meta name="robots" content="noindex"> tag in the page’s <head>.
  • X-Robots-Tag: an HTTP response header (X-Robots-Tag: noindex) which works for any response, including non-HTML files like PDFs, and can be scoped per user agent (X-Robots-Tag: googlebot: noindex).

Google respects either one. A page with noindex stays out of the index, and if it was indexed, it is eventually dropped, taking its internal links’ passage of trust with it.

The failure mode is silent

Search Console will eventually tell you a page is “Excluded by noindex tag”, but its reports run on Google’s crawl-and-index schedule, which is measured in days. Indexing changes, and the jump from “page indexed” to “dropped your noindex”, which requires a re-crawl after the new deploy, typically sits unnoticed for days.

The common scenario is not a single page going noindex by accident:

  • a staging robots.txt or noindex snippet merged into production
  • a CMS template that applies a search parameter and prints noindex sitewide (often nofollow, noindex from a bot-honeypot defaults)
  • a redirect target pointing to a previously noindexed URL
  • a cache layer serving a stale X-Robots-Tag captured from an admin route

Each of these produces a plain, legitimate page that stops ranking. No downtime alert fires, because there is no downtime.

The cheap fix: a dev team error means you find out 24 to 72 hours late

The fastest path to take the hit: check the live page and its response headers, and then check your git history. A git log -S noindex across the latest commits plus a scan of the generated HTML responses for spurious noindex is normally enough.

But the goal is to stop the 24 to 72 hours gap. The gap exists because the check is only performed when an alert is expected, which is never, for a no-op-looking deploy.

What a monitor looks like

The pipeline is mechanical: fetch the URL exactly like Googlebot would, read the raw HTML and response headers, evaluate robots.txt permissions for that specific path, and record the directives. On every later check, diff the directives you care about, such as noindex present or absent, canonical changing or title changing, then alert on semantic change rather than textual churn.

At stillindexed.com the indexability check is the first in a series of free tools. You can run it on any URL right now and see the canonical, meta robots, googlebot meta and X-Robots-Tag values it finds, plus the redirect chain, in one request:

Questions

What does noindex do?
Noindex tells a search engine not to include a URL in its index. The page can still be crawled and its links still followed unless nofollow is also present, but the page itself will not appear in search results.
Is noindex the same as blocking in robots.txt?
No, and they are not interchangeable. Robots.txt stops the fetch, noindex stops the indexing. A page blocked in robots.txt can still be listed as a bare URL, and a crawler that cannot fetch a page can never see the noindex on it.
How quickly does noindex take effect?
As soon as the page is next crawled and processed, which can be hours for a frequently crawled page. Removal is much faster than recovery, which is what makes an accidental noindex so expensive.