stillindexedStart monitoringOpen the app

Cloudflare

Cloudflare is blocking AI search crawlers: fix it

Cloudflare blocks AI crawlers by default on new domains. How to allow OAI-SearchBot, PerplexityBot and Claude-SearchBot while still blocking training.

Cloudflare asks new domains whether to allow AI crawlers, and its AI traffic controls can block Search, Agent and Training separately. Open the zone's AI bot traffic policies, keep Search allowed so OAI-SearchBot, PerplexityBot and Claude-SearchBot can reach the site, block Training, then re-check the URL with the free tool.

Why Cloudflare does this

Cloudflare owns a layer the site does not, so a crawler can be refused at the edge while the site's own files are unchanged. Since 1 July 2025, "every new domain will now be asked if they want to allow AI crawlers, giving customers the choice upfront to explicitly allow or deny AI crawlers access". Since 1 July 2026 every customer, including the Free plan, can manage AI crawlers by behaviour: Cloudflare "groups AI traffic into three behaviors you can control independently: Search, Agent, and Training", and each preset can be set to block, to block only on pages that display ads, or not to block. Search covers crawlers that index a page to answer questions about it later, Agent covers real-time fetches made on a person's behalf, and Training covers crawlers that take content to train a model. Cloudflare also notes that "a single bot can have more than one behavior", so one preset can reach a bot you meant to keep. From 15 September 2026 new domains default to blocking Training and Agent on pages that display ads, while Search remains allowed. A search bot such as OAI-SearchBot, PerplexityBot or Claude-SearchBot is removed the moment the behaviour it belongs to is set to block.

Check it right now

Before changing anything, confirm what a crawler actually sees. The check is free, takes one URL and needs no account.

Run the check

Or run the robots.txt tester against the same URL to see the exact rule that applies.

How to fix it

  1. Run the free AI crawler checker on the URL. It reports the robots.txt verdict for that path, per bot, and probes the edge with each search bot's user agent. A request that only claims to be a bot cannot reproduce IP-verified handling, so read the edge result as likely.
  2. In the Cloudflare dashboard, open the zone's Security settings and the AI bot traffic policies.
  3. Check the Search, Agent and Training presets against what you intend. As Cloudflare puts it, a bot can have more than one behaviour.
  4. Set Search to not block, so OAI-SearchBot, PerplexityBot and Claude-SearchBot can reach the site, and set Training to block. Decide Agent separately, because it governs fetches made on a person's behalf.
  5. If a WAF custom rule or the zone's managed robots.txt blocks a specific bot, fix that layer too. Cloudflare and Perplexity both document allowlisting by user agent and published IP range.
  6. Re-run the checker and fetch the live robots.txt to confirm the search bots resolve to allowed for the path.

Why it happens again

Cloudflare owns a layer the site does not: the defaults cover any behaviour nobody set explicitly, and a preset or a WAF rule can be toggled back by anyone with dashboard access. When it is, the site's own robots.txt is unchanged, every page still returns 200, and the search bots are simply refused at the edge. That is the change stillindexed watches for: it reads the robots.txt rules for the AI search crawlers on each monitored URL and raises ai_search_blocked, a warning, when a search or grounding bot that was allowed is newly disallowed. A training block raises ai_training_blocked at info and never alerts, because blocking training is usually deliberate.

stillindexed re-checks the URLs you give it every 30 minutes on Starter and alerts when a directive changes, at most 30 minutes after it does. It is a monitor rather than a crawler: it watches a list you choose and tells you when one of seven things changes. Start with a 14-day free trial, 3 sites, no card required, and a 30 day refund after that.

See what monitoring covers

Catching it next time

Fixing it once is the easy half. The setting that caused this can be changed again by anyone with access, and the page will keep returning 200 while it happens.

Other ways Cloudflare loses pages

An AI search crawler blocked from a site, on other platforms

Sources

Every claim about Cloudflare above is from their own documentation, read on 2026-10-12. Platforms change their settings; if one of these is out of date, their page wins and we would like to know.