AI crawler checker
Know when your site becomes invisible to ChatGPT search, Perplexity, Claude or Gemini, and which robots.txt line did it. This check reads the live file and answers, for each of the thirteen tracked AI crawlers, whether it may fetch your URL — naming the deciding group and rule — then probes the edge once as each of the three AI search crawlers against a browser request. It makes no model calls and ranks nothing; it is an HTTP check, exactly.
Up to five fetches: the page as a browser and as robots.txt, then one probe per AI search crawler. That is why it takes a few seconds and costs up to five of your 60 hourly checks.
Illustrative result for a reserved example domain. Your check replaces it.
OAI-SearchBot is blocked
While this rule stands, your pages cannot be crawled for those search answers. Which line to remove is below.
- robots.txt
- read
- Path evaluated
- /
Search
Build a live search index for ChatGPT search, Claude and Perplexity. Blocking one of these removes your pages from that product’s answers.
| Crawler | Access | Deciding group | Rule |
|---|---|---|---|
| OAI-SearchBotOpenAI · ChatGPT search | blocked | OAI-SearchBot | Disallow: / |
| Claude-SearchBotAnthropic · Claude search | allowed | none | no rule |
| PerplexityBotPerplexity · Perplexity search | allowed | none | no rule |
Grounding
Feeds content into a model at answer time. Google-Extended governs Gemini’s use of your content, and explicitly not Google Search or AI Overviews, which follow Googlebot.
| Crawler | Access | Deciding group | Rule |
|---|---|---|---|
| Google-ExtendedGoogle · Gemini apps grounding + Gemini training | blocked | Google-Extended | Disallow: / |
Training
Trains or fine-tunes models. Blocking these is usually a deliberate, permanent decision, so a block here is informational.
| Crawler | Access | Deciding group | Rule |
|---|---|---|---|
| GPTBotOpenAI · model training | allowed | * | no rule |
| CCBotCommon Crawl · dataset used for training | allowed | none | no rule |
User-initiated
Fetches one URL because a person asked a chat product to look at it. Not a crawler, and most operators ignore robots.txt for it.
| Crawler | Access | Deciding group | Rule |
|---|---|---|---|
| ChatGPT-UserOpenAI · ChatGPT user-initiated fetches | allowed | none | no rule |
User-initiated fetchers (ChatGPT-User, Claude-User, Perplexity-User) fetch a single URL because a person asked for it, and most operators do not treat robots.txt as applying to them. A block in the User-initiated group usually does not stop those fetches.
Edge probe: do the search crawlers get a different answer?
Each AI search crawler’s user agent, and a browser, requests the same URL once. A row marked likely blocked means the crawler-shaped request was turned away while the browser was served.
| Crawler | Bot status | Browser status | Edge verdict |
|---|---|---|---|
| OAI-SearchBot | 403 | 200 | likely blocked |
| Claude-SearchBot | 200 | 200 | same as browser |
| PerplexityBot | 200 | 200 | same as browser |
These verdicts are labelled likely on purpose. A request that only claims to be the crawler cannot reproduce the IP-verified handling a real crawler gets, so same is not proof of access and likely blocked is not proof of a block.
One check shows now
You found the current problem. Watch for the next change.
stillindexed repeats this check every 30 minutes on Starter or every 15 minutes on Agency. Critical changes are confirmed with a second fetch, then sent to Telegram and email.
- NowCurrent issue found
- Next deployA directive changes
- Next checkChange detected
- ConfirmedTelegram and email sent
A single check is a snapshot. The line that blocks an AI crawler is usually added by a “block AI bots” preset, a plugin update or a CDN setting, and it stays until someone notices the traffic is gone. stillindexed watches robots.txt and alerts on ai_search_blocked every time a search or grounding crawler that was allowed becomes disallowed.
What this check means
- AI crawler
- An AI crawler is a bot an AI operator runs to fetch your pages. Some build a live search index (OAI-SearchBot, Claude-SearchBot, PerplexityBot), some gather training data (GPTBot, ClaudeBot, CCBot, meta-externalagent, Bytespider), and some are control tokens that send no request at all (Google-Extended, Applebot-Extended).
- Product token
- A product token is the name a crawler is addressed by in robots.txt, matched exactly and case-insensitively. A group naming "Claude" does not apply to ClaudeBot or Claude-SearchBot; that prefix trap is why this tool does not use a prefix-matching parser.
- Control token
- A control token is a robots.txt-only instruction. Google-Extended and Applebot-Extended have no user agent: a robots.txt rule is their only interface, and it governs grounding and training rather than a fetch.
- Edge probe
- The edge probe requests the URL once with a crawler user agent and once as a browser, then compares the responses. It can show a bot wall refusing a crawler while serving visitors normally, without the site owner ever seeing it.
How to read the result
- Search
- The crawlers that decide whether your pages can be cited in ChatGPT search, Claude and Perplexity. A block here is the one that quietly removes the site from those answers.
- Grounding
- Google-Extended: whether Google may use crawled content for Gemini grounding and training. It does not affect Google Search or AI Overviews, which follow Googlebot.
- Training
- Crawlers that gather data to train models. Blocking these is usually a deliberate decision, so a block here is reported as information, not as a problem.
- User-initiated
- Fetchers that run because a person asked a chat product to look at one URL. Most operators ignore robots.txt for these, so a block here often has no effect.
- Deciding group and rule
- The group that governed the verdict, as written in the file ("*" for the wildcard group), and the exact line that matched. A block always names the line to go and edit.
- Unknown
- When robots.txt cannot be read (a 5xx, a timeout, a parse failure), access is reported as unknown rather than allowed. A reading we could not take is never presented as a clean answer.
Common causes
- A "block AI crawlers" preset or plugin turned on every AI token at once, including the search crawlers you wanted to keep.
- A staging robots.txt shipped to production with a broad Disallow that names an AI token.
- A CDN or WAF rule written to stop scrapers that catches the AI search crawlers too.
- A Google-Extended rule kept while the site is also being blocked from Gemini grounding, which is a supported combination and not a fault.
- A bot wall (Cloudflare, DataDome, PerimeterX) refusing OAI-SearchBot by user agent or IP while serving browsers normally.
What does it mean if OAI-SearchBot is blocked?
OAI-SearchBot is the crawler OpenAI runs so ChatGPT search can cite your pages. A Disallow rule for it stops that crawl, so your pages cannot be read into those answers, even though they stay in Google and Bing. The checker names the deciding group and the exact rule, so you can see the line to remove.
Does blocking GPTBot remove me from ChatGPT search?
No. GPTBot trains models; OAI-SearchBot serves ChatGPT search. They are separate tokens and either can be set on its own. Blocking GPTBot refuses training data while leaving the search crawler free to read your pages, which is the combination most sites want.
Does blocking Google-Extended remove me from Google Search?
No. Google-Extended is a control token, not a crawler. Blocking it stops Google using your content to train Gemini and to ground Gemini answers. Google Search and AI Overviews follow Googlebot, which Google-Extended does not touch.
Why is the edge result labelled "likely"?
The edge probe sends a request that claims to be the crawler from our own network. A real crawler is verified by its IP range, so a site that treats verified crawlers differently will answer this request differently. "Same" is not proof of access, and "likely blocked" is not proof of a block; both are evidence you can follow up in your own edge logs.
Why are the user-initiated fetchers listed at all?
ChatGPT-User, Claude-User and Perplexity-User fetch one URL because a person asked a chat product to look at it. Most operators treat robots.txt as not applying to them, so a block in that group often has no effect. They are shown for completeness, and clearly separated from the crawlers that build an index.
Does an allowed reading here mean my pages are in the AI answers?
No. This check answers one question: may the named crawler fetch this path, and does the edge treat it like a browser. Being allowed is necessary, not sufficient, for appearing in an answer. It rules out the silent failure where a robots.txt line made the site invisible, which is the one you cannot see anywhere else.