GPTBot is not ChatGPT search: which AI crawlers to block and which to allow

GPTBot is not ChatGPT search. GPTBot is OpenAI’s training crawler, and the token that puts a site into ChatGPT search answers is OAI-SearchBot. They are separate robots.txt tags with separate jobs, and OpenAI documents them as independent, so a rule that disallows GPTBot stops training use and leaves the site in search. A rule that disallows OAI-SearchBot does the opposite: it removes the site from ChatGPT search answers. Nearly every “block AI crawlers” preset treats the two as one, which is how a site ends up invisible to ChatGPT search while its owner believes it only refused training.

The short version. There are 13 tokens in the set that matters, split into four jobs: search, grounding, training, and user-initiated fetches. Blocking a search token costs citations. Blocking a training token costs nothing you can see. Blocking a grounding token costs Gemini use and not Google Search. And the -User tokens act on a person’s request rather than crawling, so a rule aimed at them is closer to blocking a visitor.

The 13 tokens and what blocking each one does

Every row but one is from the operator’s own documentation, cited at the end. ByteDance publishes nothing about Bytespider, so that row is what is commonly reported about it and nothing more than that.

Token (robots.txt)	Operator	Purpose	What blocking it does
OAI-SearchBot	OpenAI	ChatGPT search	Removes the site from ChatGPT search answers. OpenAI says it can still appear as a navigational link.
GPTBot	OpenAI	Model training	Stops training use. Does not change ChatGPT search.
ChatGPT-User	OpenAI	User-initiated fetches	OpenAI states that because the action is user-initiated, robots.txt rules may not apply.
Claude-SearchBot	Anthropic	Claude search	Reduces the site’s visibility in Claude search results, per Anthropic.
ClaudeBot	Anthropic	Model training	Excludes future content from Claude training datasets.
Claude-User	Anthropic	User-initiated fetches	Anthropic documents it as honouring robots.txt, unlike its OpenAI counterpart.
PerplexityBot	Perplexity	Perplexity search	Keeps the site out of Perplexity answers. It is not used for training.
Perplexity-User	Perplexity	User-initiated fetches	Perplexity states this fetcher generally ignores robots.txt rules.
Google-Extended	Google	Gemini grounding and training	Control token, no HTTP user agent. Blocks Gemini training and grounding. Does not affect Google Search or AI Overviews.
Applebot-Extended	Apple	Apple AI training	Control token. Blocks Apple AI training. Does not affect Applebot search.
CCBot	Common Crawl	Dataset used for training	Removes the pages from the Common Crawl archive that training datasets are built on.
meta-externalagent	Meta	Meta AI training	Blocks Meta AI training crawls.
Bytespider	ByteDance	Model training	Commonly reported as a ByteDance training crawler. ByteDance publishes no documentation for it, so this is not an operator-confirmed claim.

Two of these are not crawlers at all. Google-Extended and Applebot-Extended have no separate HTTP user agent, so there is nothing in a log file to match. They are control tokens: a robots.txt rule that tells the operator whether the content it already crawled with a normal Google or Apple crawler may be used for AI. That is why blocking Google-Extended does not remove a site from Google Search, and does not remove it from AI Overviews either, because both follow Googlebot.

The rule people get wrong

A bot with its own User-agent: group is governed by that group, not by the wildcard group. This is the one thing to understand before editing a file.

User-agent: *
Disallow: /

User-agent: OAI-SearchBot
Allow: /

The wildcard group disallows everything, and OAI-SearchBot is allowed anyway, because it has its own group. The wildcard rule does not reach a bot that named itself. The reverse is also true: if a site has no OAI-SearchBot group and the wildcard group disallows the path, OAI-SearchBot is blocked by the wildcard group and there is no line saying so.

A robots.txt that allows AI search and blocks training

One position, spelled out. Allow the search and grounding tokens, disallow the training tokens, and leave the -User tokens alone.

# Allow AI search, block training.
User-agent: *
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Applebot-Extended
Disallow: /

# Google-Extended is deliberately not listed. It refuses Gemini grounding as
# well as Gemini training, and this file keeps a site citable in Gemini apps.
# Add a "User-agent: Google-Extended" Disallow to refuse both.

Why this shape. Each token gets its own group so the wildcard Allow cannot be read as an accident, and the training tokens are named explicitly so a later edit cannot silently drop one. The -User tokens are not listed at all, which is deliberate: OpenAI says robots.txt may not apply to ChatGPT-User, and blocking a user-initiated fetch is closer to blocking a reader than filtering a crawler. One trade-off is left open on purpose. Google-Extended is not listed here, so Gemini apps can still ground answers in the content as well as train on it. Disallowing Google-Extended refuses Gemini grounding as well as Gemini training, because Google uses one token for both, so a site that wants to be cited in Gemini apps should not list it.

If you use Cloudflare, robots.txt is only half the story, because Cloudflare can refuse the request before it ever reads the file.

The Cloudflare angle

Cloudflare changed its defaults. Since 1 July 2025, new domains are asked at signup whether to allow AI crawlers, and Cloudflare’s own words are that “every new domain will now be asked if they want to allow AI crawlers, giving customers the choice upfront to explicitly allow or deny AI crawlers access”. Since 1 July 2026 every customer, including the Free plan, can manage AI crawlers by behaviour instead of one “Block AI bots” switch: Cloudflare “groups AI traffic into three behaviors you can control independently: Search, Agent, and Training”. Each preset can be set to block, to block only on pages that display ads, or not to block. From 15 September 2026, new domains default to blocking Training and Agent on ad pages while Search remains allowed.

The consequence for a site is that the same bot can be refused at two layers. A robots.txt rule that allows OAI-SearchBot still loses to an edge preset that blocks Search, and Cloudflare notes that a single bot can carry more than one behaviour. Google Search is unaffected by any of this, so a site can be missing from ChatGPT search while ranking normally in Google, which is exactly the pattern that takes days to notice.

Cloudflare verifies AI bots by IP, so a request that only claims to be a bot cannot reproduce its handling. That is why the free checker below labels the edge result as likely rather than certain.

Check it right now

Two free checks, one URL, no account:

• The AI crawler checker resolves the robots.txt verdict for the URL’s path, per bot, and probes the edge with each search bot’s user agent.
• The robots.txt tester shows the exact rule that applies to a path, so a wildcard group blocking a bot you thought you had allowed is visible as a line rather than a guess.

If the robots.txt side is fine and the site is still missing, the block is at the edge. The fix pages walk the platform specifics: Cloudflare, OAI-SearchBot and ChatGPT search, PerplexityBot, and the Google-Extended misconception.

Which line changed, and when

A robots.txt is generated by whatever serves it, and an AI-block preset, a plugin update or a dashboard toggle can rewrite the rules that decide whether the search bots are allowed. Every page still returns 200 while it happens, so nothing that watches uptime or status codes will notice. stillindexed re-checks the robots.txt rules against the AI search crawlers for each monitored URL and raises ai_search_blocked, a warning, when a search or grounding bot that was allowed is newly disallowed. A training token that gets blocked is recorded as ai_training_blocked at info and never alerts, because blocking training is usually deliberate. The point is not to argue about AI policy. It is to know which line changed, on which site, within one cycle instead of one quarter.

Sources

• OpenAI documents OAI-SearchBot, GPTBot and ChatGPT-User, including the 24 hour figure and the statement that the settings are independent, at developers.openai.com/api/docs/bots.
• Anthropic documents ClaudeBot, Claude-SearchBot and Claude-User in its crawler support article.
• Perplexity documents PerplexityBot and Perplexity-User, its WAF guidance and the 24 hour figure at docs.perplexity.ai.
• Google documents Google-Extended and its control-token status under Google’s common crawlers.
• Apple documents Applebot and Applebot-Extended, including that Applebot-Extended does not crawl pages, at support.apple.com/en-us/119829.
• Common Crawl documents CCBot and its robots.txt opt-out at commoncrawl.org/ccbot.
• Meta documents meta-externalagent and its other web crawlers at developers.facebook.com/docs/sharing/webmasters/web-crawlers.
• ByteDance publishes no documentation for Bytespider, which is why that row is marked as commonly reported rather than documented.
• Cloudflare’s change is in its changelog on AI traffic options, and the earlier default is in its 1 July 2025 press release.

Where an operator does not document a behaviour, this post says so instead of guessing.

Read next

• How AI crawlers actually see your site
• How to monitor robots.txt for changes that block Google
• Why a generic change monitor does not catch this