Netlify
Netlify _headers: where the noindex comes from
Netlify noindexes Deploy Previews and old branch deploys by header, and the _headers file is global. Which deploys get noindex and how to scope your own.
Netlify adds "X-Robots-Tag: noindex" to Deploy Previews and older branch deploys, and not to the newest branch deploy. Your own _headers file applies globally to every deploy and cannot be scoped by context, so a rule written for staging hits production too. Fix it with a context build command that copies a branch-specific header file.
Why Netlify does this
Netlify's deploy overview states: "Netlify adds an X-Robots-Tag: noindex header to Deploy Previews and older branch deploys", and "Netlify does not add this header to the most recent branch deploy for a branch so that branch deploys can be used as different versions of your app or site". Netlify also warns the header "is not a guarantee though that your preview URLs will stay protected" and recommends private projects or password protection for non-production deploys. On the custom side, the _headers docs state that headers declared in _headers or netlify.toml "are global for all builds and cannot be scoped for specific branches or deploy contexts", with a documented workaround: keep per-context header files in a folder such as custom-headers and append a copy step to each context build command, for example "[context.staging] command = \"npm run build && cp ./custom-headers/_stagingHeaders ./dist/_headers\"". A second documented limit matters for debugging: custom headers "apply only to files Netlify serves from our own backing store", so proxied content, functions and SSR pages never show them.
Check it right now
Before changing anything, confirm what a crawler actually sees. The check is free, takes one URL and needs no account.
Or run the robots.txt tester against the same URL to see the exact rule that applies.
How to fix it
- Run the X-Robots-Tag checker on the URL in question and record the exact header value.
- Identify the deploy type from the URL: "deploy-preview-" and "agent-" prefixes are Deploy Previews and get noindex from Netlify automatically; a branch-name prefix is a branch deploy.
- If production returns noindex, open _headers and netlify.toml and remove or condition the "X-Robots-Tag: noindex" rule; both files apply to every deploy.
- To noindex one branch only, create custom-headers/_stagingHeaders containing "/*" and an indented "X-Robots-Tag: noindex", then append the documented copy step to that branch context build command in netlify.toml.
- Redeploy and run the checker again on both the production URL and the branch URL.
- If SSR or proxied paths ignore the header, remember custom headers do not apply there; the function or origin behind the proxy must send it.
Why it happens again
The automatic header covers the deploy types Netlify picked, not the ones your team picked: the newest branch deploy is deliberately indexable, and a branch subdomain like staging.yourcustomdomain.com is just a branch deploy wearing a real hostname. Meanwhile every _headers edit is global, so each attempt to fix one environment re-rolls the other. Neither change sends a notice.
stillindexed re-checks the URLs you give it every 30 minutes on Starter and alerts when a directive changes, at most 30 minutes after it does. It is a monitor rather than a crawler: it watches a list you choose and tells you when one of seven things changes. Start with a 30-day free trial, 3 sites, no card required, and a 30 day refund after that.
Catching it next time
Fixing it once is the easy half. The setting that caused this can be changed again by anyone with access, and the page will keep returning 200 while it happens.
Other ways Netlify loses pages
- my Netlify staging site is indexed by Google
- my Netlify robots.txt is blocking pages that should be crawled
- The live site lands on the right URL after two hops, or the www and non-www versions disagree about which one is real.
- A Netlify site warns of an insecure connection because the Let's Encrypt certificate never provisioned or stopped renewing.
A noindex tag hiding the site from search, on other platforms
How Search Console reports it
Sources
Every claim about Netlify above is from their own documentation, read on 2026-10-25. Platforms change their settings; if one of these is out of date, their page wins and we would like to know.