X-Robots-Tag header: how to set, scope and verify it
How to set the X-Robots-Tag header in Apache, NGINX and Cloudflare, which rules it accepts, and how to verify what Googlebot actually receives.
The X-Robots-Tag header is an HTTP response header that carries indexing rules, and Google confirms the full robots meta rule set works in it: “Any rule that can be used in a robots meta tag can also be specified as an X-Robots-Tag.” You reach for it in three situations: when the response is not HTML (Google: “To block indexing of non-HTML resources, such as PDF files, video files, or image files, use the X-Robots-Tag response header instead.”), when you need to control an entire set of URLs at the server or CDN level, and when you want the rule enforced on staging environments where templates are hard to touch. This guide covers the rules, the server and CDN recipes, and how to verify the header Googlebot actually receives. If you want the meta tag side of the comparison first, that is covered separately in X-Robots-Tag vs meta robots.
The rules the header accepts
Every rule below works in both the X-Robots-Tag header and the robots meta tag. Meanings are one line each:
| Rule | Meaning |
|---|---|
noindex |
Do not show the page, media, or resource in search results. |
nofollow |
Do not follow the links on the page. |
none |
Equivalent to noindex, nofollow. |
all |
No restrictions; the default, no effect when written out. |
nosnippet |
Removes the text snippet and video preview; the content is also excluded as a direct input for AI Overviews and AI Mode. |
indexifembedded |
Only has an effect when accompanied by noindex. |
max-snippet:[n] |
Snippet character limit; 0 is equivalent to nosnippet, -1 lets Google choose, an unparsable value is ignored. |
max-image-preview:[setting] |
Accepts none, standard or large; large shows a preview up to the width of the viewport. |
max-video-preview:[n] |
Video snippet length in seconds; 0 allows at most a static image, -1 means no limit. |
notranslate |
Do not offer translation of the page in search results. |
noimageindex |
Do not index images on the page. |
unavailable_after |
Do not show the page in search results after the given date and time. |
Google ignores three rules you still see in the wild. noarchive, nocache and nositelinkssearchbox sit under the line “The following rules aren’t used by Google Search and are ignored.” Rule names are case-insensitive, and so are the header name and the user agent token, so casing never changes the outcome.
One related trap worth stating plainly: noindex inside robots.txt is not supported by Google. The only supported carriers for indexing directives are the meta tag and this header. If you are weighing disallow against noindex, that tradeoff is laid out in noindex vs disallow.
Scoping rules to specific bots
The header accepts a user agent token before the rules. Google’s own examples are X-Robots-Tag: googlebot: nofollow and X-Robots-Tag: otherbot: noindex, nofollow. Three details matter when you go beyond plain noindex:
- Rules without a user agent apply to every crawler. Rule names, header values and the user agent token are all case-insensitive.
- In the meta tag, Google supports only the
googlebotandgooglebot-newstokens; other values there are ignored. The header form is what you have when you need to differentiate more broadly. - A broad
noindexcovers search crawlers. Google notes that non-search crawlers such as AdsBot may need their own targeted rules if you want them excluded too.
You can send several X-Robots-Tag headers in one response, or one header with a comma-separated list. Google’s examples combine X-Robots-Tag: noimageindex with X-Robots-Tag: unavailable_after: 25 Jun 2010 15:00:00 PST. When rules conflict, the stricter one wins: “In the case of conflicting robots rules, the more restrictive rule applies.” When different crawlers are given different rules, Google adds the restrictions together: “the search engine will use the sum of the negative rules”, so a general nofollow plus a googlebot: noindex becomes noindex, nofollow for Googlebot.
Bing documented the header form before the Bing brand existed. A Microsoft Live Search blog post from June 3, 2008 says: “The tags below can be present as Meta Tags in the page HTML or X-Robots-Tag Tags in the HTTP Header,” and that when both forms are present “the most restrictive version applies.” Bing’s current help pages could not be read for this article, so treat that post as historical documentation, not a statement about today’s Bing behaviour. Google also notes that other search engines may treat these rules differently.
unavailable_after needs a valid date
unavailable_after removes a URL from results after a timestamp, which makes temporary content (a live stream, an expiring promotion) clean itself up. The date format is where implementations break. Google accepts “a widely adopted format including, but not limited to RFC 822, RFC 850, and ISO 8601”, and its documented example value is simply unavailable_after: 2020-09-21. So both of these are acceptable:
X-Robots-Tag: unavailable_after: 2027-10-01
X-Robots-Tag: unavailable_after: 25 Jun 2027 15:00:00 PST
Two behaviours to plan around. First, a malformed date means the rule is ignored: “The rule is ignored if no valid date/time is specified.” Second, the URL keeps being crawled after the date, but less: “Googlebot will decrease the crawl rate of the URL considerably after the specified date and time.”
Apache: .htaccess and virtual host recipes
Apache’s Header directive from mod_headers carries the header. It works in server config, virtual host, directory and .htaccess context (with AllowOverride FileInfo), and onsuccess is the default condition. Google’s documented example for PDFs:
<Files ~ "\.pdf$">
Header set X-Robots-Tag "noindex, nofollow"
</Files>
Single file, per Google’s example: <Files "unicorn.pdf">, with the note that the .htaccess file must sit in the directory of the matched file. Images work the same way with <Files ~ "\.(png|jpe?g|gif)$"> and Header set X-Robots-Tag "noindex".
The staging recipe is the common real-world one: point a Header set X-Robots-Tag "noindex, nofollow" at the whole virtual host and every response that host produces is tagged, templates included. If staging is your concern, the full checklist lives in staging site indexed by Google.
Two Apache details that trip people up:
- Error responses need
always. Without it, headers are only added on success. For a locally generated non-2xx response such as a redirect, only thealwaystable is applied, soHeader always set ...is what you want when 404s and redirects must also carry the rule. - Use
set, notadd, for a fixed value.Header setreplaces any previous header of that name;Header addcan leave two or more headers with the same name in the response. Duplicate headers are exactly the kind of thing a verification check flags.
NGINX: location blocks and the add_header inheritance trap
NGINX’s add_header directive has context http, server, location, if in location, and Google’s own NGINX example for PDFs is:
location ~* \.pdf$ {
add_header X-Robots-Tag "noindex, nofollow";
}
Two behaviours you must know before writing your config:
- Status codes. Without
always, the header is added only when the response code is 200, 201, 204, 206, 301, 302, 303, 304, 307 or 308. Withalways, “the header field will be added regardless of the response code.” If a noindex must also cover 404s (a common staging setup),alwaysis required. - Inheritance. The nginx docs state: “These directives are inherited from the previous configuration level if and only if there are no add_header directives defined on the current level.” Read that twice. If your server block already adds security headers and you add one
add_headerin a location block for noindex, that location silently loses the security headers. The fix is to re-declare every header in that block, or, on NGINX 1.29.3 and later, useadd_header_inherit, where themergeparameter appends values from the previous level to those defined at the current level.
The result is a response where the noindex works on the URLs you targeted and the headers that used to be there are gone, with nothing in the error log to say so.
Cloudflare: transform rules and the Pages _headers file
On a proxied zone, a Response Header Transform Rule can set, add or remove response headers. A “set” operation overwrites an existing header of that name or adds it if it is missing. The dashboard path starts at “Select Create rule > Response Header Transform Rule”, and a single rule can modify up to 30 headers. Rules run in order, so a later rule can overwrite what an earlier one did. Two limits: you cannot add, modify or remove headers whose name starts with cf- or x-cf-, and the rules also apply to Cloudflare’s default error pages and Custom Errors, which is relevant if you want the header on error responses.
On Cloudflare Pages, the header lives in a plain text file named _headers with no extension, placed in the static asset directory. Cloudflare’s own example noindexes every pages.dev host:
https://:project.pages.dev/*
X-Robots-Tag: noindex
https://:version.:project.pages.dev/*
X-Robots-Tag: noindex
Limits and gotchas of _headers: up to 100 header rules, a 2,000 character limit per line, and duplicate applications of a header join their values with a comma, producing results like X-Robots-Tag: nosnippet, noindex. Two behaviours matter for indexing work: redirects take priority over headers (“when a request matches both a redirect and a header, the redirect takes priority”), and _headers rules are “not applied to responses generated by Pages Functions”, which includes SSR responses. A Next.js or Astro site on Pages with server-rendered routes will not get the header from _headers; you need the framework itself or a Worker to tag those.
Verify what Googlebot actually receives
Config that compiles is not config that ships the right header. Two curl forms cover verification:
curl -sI https://example.com/downloads/brief.pdf | grep -i x-robots-tag
curl -sI -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \
https://example.com/downloads/brief.pdf | grep -i x-robots-tag
The second matters because CDNs and WAFs can treat crawler user agents differently, and a bot-specific X-Robots-Tag: googlebot: noindex only proves itself when you send a Googlebot user agent. Without a terminal, the X-Robots-Tag checker reads the response headers of a URL, and the crawler diff checker compares what Googlebot and a regular browser get from the same URL, which catches exactly this class of discrepancy. For headers, status code and meta tag in one view, use the indexability checker.
One blind spot to know before you verify: robots.txt. “If a page is disallowed from crawling through the robots.txt file, then any information about indexing or serving rules will not be found and will therefore be ignored.” A URL blocked in robots.txt never shows its X-Robots-Tag to Google, and Search Console help notes that for such URLs “Indexing allowed? will always be “Yes” because Google can’t see and respect any noindex directives.” If the header does not show up at all on an error response, check for a missing always in the NGINX or Apache config.
In Search Console, a noindex that Google has seen moves the URL into the Excluded by ‘noindex’ tag status of the Page indexing report, and URL Inspection shows the same for a single URL. Google is direct about timing: “Depending on the importance of the page on the internet, it may take months for Googlebot to revisit a page.” Verification therefore has two halves: prove the header right after deploy, and then confirm the status change in Search Console over the following weeks.
The header can also change later without anyone touching it on purpose: a server config merge, a CDN rule edit, a framework upgrade. stillindexed re-checks up to 25 monitored URLs per site every 15 or 30 minutes, headers included, and alerts by email and Telegram when a noindex appears or disappears. The trial is 30 days, 3 sites, no card required, and plans are on the pricing page. If you are weighing it against Little Warden, the better-known per-URL monitor, the comparison page prices both from their published pages (October 2026).
Sources
- Google, Robots meta tag, data-nosnippet, and X-Robots-Tag specifications: rules, bot-specific forms, conflicts, Apache and NGINX examples.
- Google, Block Search indexing with noindex: revisit timing, noindex in robots.txt unsupported.
- Search Console Help, URL Inspection tool.
- nginx, ngx_http_headers_module:
always, inheritance,add_header_inherit. - Apache, mod_headers:
setvsadd,onsuccessvsalways. - Cloudflare, Response header transform rules and creating them in the dashboard.
- Cloudflare, Pages headers: the
_headersfile. - Microsoft Live Search, Robots Exclusion Protocol: joining together to provide better documentation, 3 June 2008.
Read next
Questions
- What is the X-Robots-Tag header?
- The X-Robots-Tag is an HTTP response header that carries the same indexing rules as a robots meta tag. Google states that any rule usable in a robots meta tag also works as an X-Robots-Tag, including noindex and unavailable_after. It is how you send indexing rules for PDFs, images and other files that have no HTML to hold a meta tag.
- How do I set X-Robots-Tag noindex in NGINX?
- Add add_header X-Robots-Tag "noindex"; inside the location block that serves the URLs, and add the always flag if you also want the header on error responses. Keep NGINX's inheritance rule in mind: a child block with its own add_header lines stops inheriting add_header directives defined at the parent level, so re-declare any headers you need on that block too.
- Where do I set X-Robots-Tag in Cloudflare?
- Two places. On a proxied zone, select Create rule and then Response Header Transform Rule in the Cloudflare dashboard, match the URLs, and set the header value. On Cloudflare Pages, place a plain text file named _headers in your static asset directory. Note that _headers rules are not applied to responses generated by Pages Functions, so SSR responses stay untagged.
- Does the X-Robots-Tag work on PDFs?
- Yes, and Google recommends it: to block indexing of non-HTML resources such as PDF, video or image files, use the X-Robots-Tag response header. In Apache, scope the rule with a Files directive in .htaccess. In NGINX, use a regex location block. A meta tag cannot control a PDF because there is no HTML to put it in.
- Why doesn't Google see my X-Robots-Tag header?
- If robots.txt disallows crawling of the URL, Google never fetches the response and never reads its header. Google's documentation states that indexing or serving rules will then be not found and therefore ignored. Remove the disallow first so Google can fetch the page, then check the header with curl and watch the noindex status in Search Console.
- How long until X-Robots-Tag noindex takes effect?
- It takes effect when Google next crawls the URL, which is not immediate. Google states that, depending on the importance of the page on the internet, it may take months for Googlebot to revisit a page. Until that recrawl, the URL can keep appearing in results. Verify the header right after deploy, then watch the Page indexing report.