How AI crawlers actually see your site
What GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot, Google-Extended, Applebot-Extended and CCBot are each for, what they obey, and what blocking one costs you.
The bots listed in your logs as GPTBot, ClaudeBot or PerplexityBot are not one thing. They split into three jobs: training a model on your content, retrieving your page live so it can be cited in an answer, and acting on an explicit request from a person. The job decides what blocking the bot costs you. This post walks each token: what it is for, what it obeys, and what blocking it gives up.
The short version before the details. Training crawlers feed models, retrieval crawlers feed answers, and the two do not share a token. Blocking a training crawler costs you nothing in visibility. Blocking a retrieval crawler costs you citations when a question matches your pages. The -User tokens act when a person asks the assistant to open a page, and the two vendors treat robots.txt differently there, which is worth reading carefully before you write a rule. Google-Extended and Applebot-Extended are not crawlers at all.
What are the three jobs, exactly
A training crawler visits pages in bulk to build or improve a model. Your content becomes part of the model, and the model answers without naming you. A retrieval crawler visits one page at a time, when a user has asked a question and the assistant is looking for sources. The answer can cite you, with a link back. An on-demand token acts when the user explicitly tells the assistant to open a page, like telling a browser to visit it.
The distinction matters because the two blocking decisions point opposite ways. To stop being training data, block the training token. To earn citations, keep the retrieval token open.
GPTBot
Token: User-agent: GPTBot. This is OpenAI’s training crawler. It reads pages in bulk to build the models behind ChatGPT and other products.
Blocking GPTBot costs you training presence. It does not cost you citations, because citations come from the retrieval tokens below. If you block GPTBot, your pages simply are not used as training material. Whether GPTBot renders JavaScript before reading a page is not publicly documented.
OAI-SearchBot
Token: User-agent: OAI-SearchBot. This is OpenAI’s retrieval crawler. It visits a page when a user asks a question and the assistant is looking for sources, so the answer can cite you. Blocking OAI-SearchBot costs you citations in ChatGPT search answers. It does not stop training use.
ChatGPT-User
Token: User-agent: ChatGPT-User. This token acts when a person explicitly asks ChatGPT to open a specific page or follow a link. OpenAI’s own documentation says that “because these actions are initiated by a user, robots.txt rules may not apply”. Treat a rule here as a request rather than a control, and understand that blocking it is closer to blocking that person’s visit than to stopping a crawler.
ClaudeBot
Token: User-agent: ClaudeBot. Anthropic’s training crawler. It reads pages in bulk for Claude model training. Blocking ClaudeBot costs training data, not citations. JavaScript rendering for ClaudeBot is not publicly documented.
Claude-User
Token: User-agent: Claude-User. This acts when a person explicitly asks Claude to open a page. It is worth separating from ChatGPT-User rather than assuming the two behave alike: Anthropic documents Claude-User as honouring robots.txt along with its other agents, where OpenAI says robots.txt may not apply to ChatGPT-User. Same job, different published commitment.
Claude-SearchBot
Token: User-agent: Claude-SearchBot. This is Claude’s retrieval token, used when answering a question with live sources. Blocking Claude-SearchBot means you are not cited in Claude answers. It does not change training use.
PerplexityBot
Token: User-agent: PerplexityBot. Perplexity’s retrieval crawler, used when a user asks a question and the answer draws on live pages with citations. Blocking PerplexityBot costs you citations in Perplexity answers. Whether it executes JavaScript is not publicly documented.
Google-Extended
Token: User-agent: Google-Extended. This is not a crawler. Google-Extended is an opt-out token that controls whether content already crawled by Googlebot can be used to train AI models and related services. Blocking Google-Extended does not stop Googlebot and does not change your search ranking. Google states this in its own crawler documentation: “Google-Extended does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search.” This is the most commonly misunderstood entry in the list, and the correction matters: you can block training use without touching search.
Applebot-Extended
Token: User-agent: Applebot-Extended. Same shape as Google-Extended. It is not a crawler. It is an opt-out token for Apple’s model training, and blocking it does not affect Applebot’s search crawling or anything a user sees in Apple search.
CCBot
Token: User-agent: CCBot. This is Common Crawl’s crawler, which archives a public copy of the web. That archive is used to train a range of models you never meet directly. Blocking CCBot removes your pages from the Common Crawl archive, which removes them from the training data built on it. The effect on your citations is indirect, because you are never cited by the archive itself.
robots.txt is per-token, and the most specific group wins
A bot with its own User-agent: group is governed by that group, not by the star group. This is the rule readers get wrong most often:
User-agent: *
Disallow: /
User-agent: GPTBot
Allow: /The star group disallows everything, but GPTBot has its own group, so GPTBot is allowed. The star rule does not apply to a bot that has its own group. To block GPTBot in this file you must list it in a group that disallows it:
User-agent: *
Disallow: /
User-agent: GPTBot
Disallow: /The worked file: three positions
Position one, allow everything for all of them:
User-agent: *
Allow: /Position two, allow retrieval but refuse training. Open the retrieval tokens, block the training tokens and the archive:
User-agent: *
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /Position three, refuse everything:
User-agent: *
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-User
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: CCBot
Disallow: /One caveat on position three: ChatGPT-User can still appear, because OpenAI says robots.txt may not apply when a person asked for the page, and Google-Extended and Applebot-Extended are opt-out markers rather than something that visits you. Position three is the strictest thing you can write in robots.txt, and no further. To make it clear whether the bot is served the same page as the browser, run a URL through the crawler diff checker at /tools/crawler-diff-checker, which fetches as Googlebot, Bingbot, GPTBot and a browser and reports where the responses differ.
Where these facts come from
Every behaviour above is taken from the operator’s own documentation rather than from observation, because operators change what their bots do and only the documentation says what they intend. Check these directly before you write a rule, they are short pages:
- OpenAI publishes GPTBot, OAI-SearchBot and ChatGPT-User at developers.openai.com/api/docs/bots.
- Anthropic publishes ClaudeBot, Claude-User and Claude-SearchBot in its crawler support article.
- Google documents Google-Extended in its list of common crawlers.
Where a behaviour is not documented, this post says so instead of guessing.
Questions
- What is the difference between a training crawler and a retrieval crawler?
- A training crawler reads pages to build a model. Blocking it means your content is not used as training data. A retrieval crawler reads pages when a user asks a question and you can be cited in the answer. Blocking it means you are not cited, which is the acquisition you lose. The two have opposite consequences.
- Does blocking Google-Extended affect my search ranking?
- No. Google-Extended is an opt-out token, not a crawler. It controls whether content already crawled by Googlebot can be used to train AI models. Blocking it does not stop Googlebot and does not change how your pages rank in search.
- Does a robots.txt Disallow for User-agent star apply to a named bot?
- No. The most specific group wins. A bot with its own User-agent group is governed by that group, not by the star group. To block a bot that has its own group, you must list that bot's token in its own group.
- Can I block ChatGPT-User or Claude-User with robots.txt?
- They differ, so check each one. OpenAI documents ChatGPT-User as user-initiated and states that "because these actions are initiated by a user, robots.txt rules may not apply". Anthropic documents Claude-User as honouring robots.txt like its other agents. Both act when a person asks the assistant to open a page, so blocking either is closer to blocking a visitor than to blocking a crawler.
- Which AI crawlers respect robots.txt?
- GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Claude-User, CCBot, Googlebot and Applebot are all documented by their operators as honouring robots.txt. ChatGPT-User is the exception worth knowing: OpenAI states that robots.txt rules may not apply to it, because the visit is initiated by a person.
- Does blocking a training crawler stop it from being cited?
- Only if the same token is also used for retrieval. Training and retrieval are different functions with different tokens. Blocking GPTBot stops training use. Blocking OAI-SearchBot stops live retrieval in ChatGPT search answers. Decide separately for each.