Cloudflare Pages
Cloudflare Pages previews: the noindex header
Preview deployments carry X-Robots-Tag: noindex by default, but branch aliases on custom domains may not. The pages.dev versus custom domain trap.
Cloudflare Pages adds "X-Robots-Tag: noindex" to every preview deployment by default, but a branch alias attached to a custom domain and the *.pages.dev production host need their own checks. Verify the header per URL, add a _headers rule for any host missing it, and point canonicals and the sitemap at the production domain only.
Why Cloudflare Pages does this
Cloudflare's Pages documentation: "By default, every preview deployment generated by Cloudflare Pages includes the X-Robots-Tag: noindex HTTP response header," on the hash-based URLs like 373f31e2.project.pages.dev. Three documented gaps sit around that default. Preview aliases: Pages also creates a branch alias such as development.project.pages.dev, and a branch can be "attach[ed]" to a custom domain, which serves preview content on an indexable host you own. Access: "By default, these deployment URLs are public" unless you enable an Access policy, which covers preview URLs but "not your *.pages.dev domain or custom domain." And duplication: the production *.pages.dev subdomain serves the same content as the custom domain, so if both are crawled, the canonical choice belongs to your tags and sitemap, not to Pages.
Check it right now
Before changing anything, confirm what a crawler actually sees. The check is free, takes one URL and needs no account.
Or run the robots.txt tester against the same URL to see the exact rule that applies.
How to fix it
- Run the X-Robots-Tag checker on the preview URL that appeared in Google. A hash-based preview should return the noindex header; if it does, no platform change is needed.
- Check any branch alias or custom domain attached to a non-production branch. If it lacks the header, add a _headers file rule for that host: the host pattern, then "X-Robots-Tag: noindex".
- Decide which host is canonical. If production content should live only on the custom domain, keep the *.pages.dev production URL out of internal links, sitemaps and canonical tags.
- Enable a Cloudflare Access policy on preview deployments if previews should not be public at all: Workers & Pages > project > Settings > General > Enable access policy.
- Redeploy and re-run the checker on every host variant until only the production domain returns an indexable response.
- If Google already lists a preview host, use Search Console removals after the header is live.
Why it happens again
Every pull request mints a new public preview URL and every branch alias follows the branch, so the surface area grows with the team. The _headers file is edited on redesigns and a rule scoped too narrowly stops covering the hosts it was written for, while the production pages.dev twin keeps serving duplicate content quietly in the background.
stillindexed re-checks the URLs you give it every 30 minutes on Starter and alerts when a directive changes, at most 30 minutes after it does. It is a monitor rather than a crawler: it watches a list you choose and tells you when one of seven things changes. Start with a 30-day free trial, 3 sites, no card required, and a 30 day refund after that.
Catching it next time
Fixing it once is the easy half. The setting that caused this can be changed again by anyone with access, and the page will keep returning 200 while it happens.
A staging or preview site indexed by Google, on other platforms
How Search Console reports it
Sources
Every claim about Cloudflare Pages above is from their own documentation, read on 2026-10-25. Platforms change their settings; if one of these is out of date, their page wins and we would like to know.