ChatGPT search stopped listing your site: OAI-SearchBot

Why OpenAI does this

OpenAI's crawler documentation separates the two jobs, and the separation is the whole answer. "OAI-SearchBot is used to surface websites in search results in ChatGPT's search features. Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links." GPTBot is the training crawler: "Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models." OpenAI states plainly that the settings are independent: "a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training." It also warns that "it can take ~24 hours from a site's robots.txt update for our systems to adjust". The usual cause is a rule aimed at training that also names the search token: a security plugin or an AI-block preset writes a Disallow for a list of AI user agents and OAI-SearchBot is on the list. A plain "User-agent: *" group that disallows the path does the same thing when no OAI-SearchBot group exists, because the star group is what governs the bot.

Check it right now

Before changing anything, confirm what a crawler actually sees. The check is free, takes one URL and needs no account.

Run the check

Or run the robots.txt tester against the same URL to see the exact rule that applies.

How to fix it

1. Run the free AI crawler checker on the URL to see the robots.txt verdict for OAI-SearchBot and GPTBot on that path.
2. Fetch the live robots.txt, or run the robots.txt tester, and find the group that governs OAI-SearchBot: its own group if one exists, otherwise the star group.
3. If the star group disallows the path, give OAI-SearchBot its own group with "Allow: /" so it is no longer caught by the star rule.
4. Keep any GPTBot Disallow you want. It is a separate token and does not affect ChatGPT search.
5. Save and republish, then confirm the live response changed. A CMS, SEO plugin or CDN can serve its own copy of robots.txt, so the editor is not proof.
6. Re-check with the tool. OpenAI states it can take about 24 hours from a robots.txt update for its systems to adjust.

Why it happens again

The file is generated by whatever serves it, so the rule can be rewritten by a plugin update, a theme switch or an AI-block preset, none of which touch the pages themselves. The site keeps returning 200 and the search bot is quietly disallowed, which is why the loss does not show up until someone notices the citations are gone. stillindexed watches the rules that matter here: it reads the AI crawler rules for each monitored URL and raises ai_search_blocked, a warning, when a search or grounding bot that was allowed is newly disallowed. A change to a training token is recorded as ai_training_blocked and never alerts, so GPTBot noise does not page you.

stillindexed re-checks the URLs you give it every 30 minutes on Starter and alerts when a directive changes, at most 30 minutes after it does. It is a monitor rather than a crawler: it watches a list you choose and tells you when one of seven things changes. Start with a 14-day free trial, 3 sites, no card required, and a 30 day refund after that.

See what monitoring covers

Catching it next time

Fixing it once is the easy half. The setting that caused this can be changed again by anyone with access, and the page will keep returning 200 while it happens.

• Six checks when a site disappears from Google
• Why a generic change monitor does not catch this

An AI search crawler blocked from a site, on other platforms

• Cloudflare
• Perplexity
• Google

Sources

Every claim about OpenAI above is from their own documentation, read on 2026-10-12. Platforms change their settings; if one of these is out of date, their page wins and we would like to know.

• https://developers.openai.com/api/docs/bots
• https://developers.google.com/search/docs/crawling-indexing/robots/intro