X-Robots-Tag checker

Read the crawler directives in the final HTTP response header, grouped by user agent, then compare them with the robots values in the HTML.

Public check, no account needed. Reads the X-Robots-Tag header from the final response, scoped per user agent. Rate limited to 60 an hour per IP.

Worked example

Illustrative result for a reserved example URL. Your check replaces it.

Blocked by X-Robots-Tag

Noindex applies to googlebot.

X-Robots-Tag, googlebot: noindex, nofollow
HTTP status: 200
Final URL: https://example.com/pricing
Meta robots: not found
Googlebot meta: not found

What this check means

X-Robots-Tag: An X-Robots-Tag is an HTTP response header that tells a crawler whether it may index content or follow links.
User-agent scope: User-agent scope limits a directive to a named crawler, while the asterisk scope applies it to every crawler that supports the header.
Noindex: Noindex is a crawler directive that prevents the covered URL from being included in a search index.

How to read the result

Verdict: Blocked means at least one returned header group contains noindex or none, or the final response itself failed.
Header by agent: Each row names the crawler scope and its directives. A Googlebot row affects Google even when the all-agents row is clean.
HTTP status: This is the status of the final response whose headers were inspected. A 4xx or 5xx response needs attention before its directives matter.
Final URL: This is the destination after redirects. It confirms which response supplied the displayed header.
Meta robots and Googlebot meta: These HTML values provide the comparison. A noindex in either the header or the matching HTML tag is enough to block indexing.

Common causes

A web server rule adds noindex to a file type, directory, or status response.
A reverse proxy or CDN injects a header that application templates cannot see.
A staging header policy remains active after a production deployment.
Crawler-specific middleware targets Googlebot while general browser requests look clean.
A redirect moves the request onto a host with a different header policy.

Where is the X-Robots-Tag found?

It is found in the HTTP response headers returned by the server. It is not part of the HTML document, so viewing the page source will not reveal it.

Can X-Robots-Tag target only Googlebot?

Yes. A header can scope directives to a named crawler such as Googlebot while leaving the general directives unchanged. The result groups returned directives by user agent.

Which wins if the header and meta robots disagree?

A restrictive instruction from either source still applies. If one says index and the other says noindex, the page remains blocked for the crawler covered by noindex.

Why can a browser extension miss this noindex?

Extensions that inspect only the document can miss response headers, especially after redirects. This checker reads the headers from the final response and shows the HTML directives beside them.