Robots.txt tester
A robots.txt tester resolves whether a crawler is allowed to fetch a given path. This one reads the live file and answers per path, for the wildcard group and for Googlebot separately, because those two often disagree.
Illustrative result for a reserved example domain. Your check replaces it.
1 path is blocked for Googlebot
Googlebot will not crawl these paths while this rule stands.
| Path | User-agent: * | Googlebot |
|---|---|---|
| / | allowed | allowed |
| /pricing | allowed | allowed |
| /admin | disallowed | disallowed |
| /cart | disallowed | allowed |
- robots.txt
- found, HTTP 200
- Sitemap directive
- https://example.com/sitemap.xml
Show the file we fetched
User-agent: * Disallow: /admin Disallow: /cart User-agent: Googlebot Disallow: /admin Sitemap: https://example.com/sitemap.xml
What this check means
- robots.txt
- Robots.txt is a plain text file at the root of a domain that tells crawlers which paths they may fetch.
- Effective permission
- Effective permission is the answer a specific crawler reaches for a specific path after every matching rule is applied in order.
- User-agent group
- A user-agent group is a block of rules that applies to the crawlers it names. A crawler obeys the most specific group naming it and ignores the others.
- Sitemap directive
- A Sitemap directive is a line in robots.txt that points crawlers at a sitemap file. It is independent of the allow and disallow rules.
How to read the result
- Verdict
- A path blocked for Googlebot is marked critical, because that is the crawler that affects your search traffic. A path where the wildcard group and Googlebot disagree is marked for review.
- User-agent: *
- This is the answer for a crawler that matches only the wildcard group. Most crawlers other than the big search engines land here.
- Googlebot
- This is the answer Googlebot itself reaches. When robots.txt contains a Googlebot group, this column follows that group alone.
- robots.txt status
- A 404 is a valid answer and means nothing is disallowed. Any other non-200 status means the rules cannot be resolved, and the check says so rather than guessing.
Common causes
- A staging Disallow rule shipped to production with a deploy.
- A CMS or plugin rewrote robots.txt during an update.
- A Disallow was added to the wildcard group while a Googlebot group already existed, so Googlebot never saw the rule.
- A trailing rule blocks a path prefix wider than intended, for example Disallow: /b matching /blog.
- A CDN or WAF serves a different robots.txt than the origin.
Does a Disallow rule remove a page from Google?
Not directly. Disallow stops a crawler from fetching the page. A page already in the index can stay there, listed without a description, because the crawler can no longer read it. To remove a page, allow the crawl and serve a noindex directive instead.
Why do the wildcard group and Googlebot give different answers?
A crawler obeys the single most specific group that names it and ignores the rest. If a robots.txt has both a "User-agent: *" group and a "User-agent: Googlebot" group, Googlebot follows only the Googlebot group. Rules you added to the wildcard group do not apply to it.
Does robots.txt block a page that is linked from elsewhere?
It blocks the fetch, not the link. Google can still list a disallowed URL it found through links, and because it cannot crawl the page it has no title or description to show.
How quickly does a robots.txt change take effect?
Google caches robots.txt for up to 24 hours, so a change can take a day to be picked up. That delay works both ways: an accidental Disallow keeps blocking after you revert it, and a fix is not instant.
Which paths should I test here?
Test the paths that earn traffic: your home page, main category or product paths, and your blog. Those are the URLs where an accidental Disallow costs the most and gets noticed the latest.