X-Robots-Tag vs meta robots: the noindex you cannot see in the HTML
X-Robots-Tag and meta robots both send noindex, but one lives in the response headers and one lives in the HTML. What each one does, which wins when both are present, and how to check both.
X-Robots-Tag and meta robots are the two ways to tell a search engine a page must not be indexed. Meta robots is a tag in the HTML; X-Robots-Tag is an HTTP response header. A noindex from either one blocks indexing, and a page that carries both is blocked twice over.
If you manage a site and only check the HTML, you can miss half of the story. Run the page through the indexability checker to read both sources in one request, or the X-Robots-Tag checker for the header alone.
What is meta robots?
Meta robots is an HTML tag in a page’s head that tells crawlers how to treat that page. It lives in the markup, so it is visible in view-source:
<meta name="robots" content="noindex">
It is set per page, in the page itself. If a page carries no meta robots tag, it carries no meta robots instruction at all.
What is an X-Robots-Tag?
An X-Robots-Tag is an HTTP response header that tells crawlers how to treat the response. It is not in the page; the server sends it alongside the page:
X-Robots-Tag: noindex
The header works for any file type, including PDFs and images, and can be set at the server, CDN, or edge level. It can also be scoped per agent: X-Robots-Tag: googlebot: noindex blocks only Googlebot and leaves other crawlers alone.
Which one wins when both are present?
A page is not indexed if either source says noindex. The two directives are not alternatives; the page has to satisfy both.
Consider a page with <meta name="robots" content="noindex"> and an X-Robots-Tag: noindex on the same response. It is blocked twice over, and removing only the meta tag changes nothing while the header still fires. When they conflict, the stricter directive applies, so in practice: find and clear every source that says noindex.
Why can’t I see the X-Robots-Tag in view-source?
View-source shows the HTML body and head, not the HTTP response headers the server sends with the page. The header is delivered before the HTML starts, so it never appears in the markup you inspect.
That is why a silent noindex hides: the page looks clean in the source while the response still blocks indexing. The noindex can be invisible in the page and still fully enforced.
How do I check for each?
Check the header with curl:
curl -sI https://example.com/pricing
Read the output and look for x-robots-tag: noindex. For the meta tag, view-source the page and look for <meta name="robots" content="noindex">.
Or skip curl entirely. The X-Robots-Tag checker reads the response header, and the indexability checker reads both meta and header in one request. Most free checkers read the HTML only.
When should I use one instead of the other?
Use meta robots for simple per-page control in the HTML. It is easy to read, easy to audit, and it lives with the content it governs.
Use X-Robots-Tag when you need sitewide or edge-level control, non-HTML files, or per-agent scoping. A header set once at the edge covers every response that passes through it, including files that have no HTML to carry a meta tag.
A silent noindex can live for days before the traffic chart shows it. stillindexed.com monitors your pages’ meta robots and X-Robots-Tag and alerts you when a directive appears. Starter checks every 30 minutes, Agency every 15: https://app.stillindexed.com/auth/request
Questions
- What is the difference between X-Robots-Tag and meta robots?
- They carry the same directives and differ only in where they live. Meta robots is an HTML tag in the head of the document. X-Robots-Tag is an HTTP response header. Because the header is not part of the document, it applies to any file type, including PDFs and images, which a meta tag cannot reach.
- Which one wins if they disagree?
- The more restrictive instruction applies. If the header says noindex and the HTML says index, the page is not indexed. There is no precedence rule that lets a permissive directive override a restrictive one.
- Can X-Robots-Tag target a specific crawler?
- Yes. Prefixing the value with a user agent, such as "googlebot: noindex", scopes it to that crawler and leaves others unaffected. This is why a page can look fine in a generic check and still be blocked for Google.
- Where does an unexpected X-Robots-Tag usually come from?
- A web server rule applied to a file type or directory, a reverse proxy or CDN injecting a header the application never sees, or a staging header policy left switched on after a production deploy.