Perplexity
PerplexityBot blocked: the site is missing from Perplexity
PerplexityBot surfaces pages in Perplexity answers and does not train models. A robots.txt or WAF block removes you; Perplexity-User ignores robots.txt.
PerplexityBot is the crawler Perplexity uses to surface and link sites in its answers, and it is not used to train models. A robots.txt Disallow for PerplexityBot, or a WAF rule that refuses it, keeps the site out of those results. Allow PerplexityBot in both places, then allow up to 24 hours for the change.
Why Perplexity does this
Perplexity's crawler documentation is explicit about what is lost: "PerplexityBot is designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models." Blocking it costs the citation, not a training dataset. Two layers can cause the block. The first is robots.txt, where a broad "User-agent: *" Disallow or a rule naming PerplexityBot stops the crawler. The second is a WAF, which fits a request pattern rather than reading robots.txt. Perplexity documents that case directly: create a custom rule that allows requests whose User Agent contains PerplexityBot or Perplexity-User and whose IP source is in its published ranges, and set the action to Allow so it takes priority. Perplexity also states that changes "may take up to 24 hours for our systems to reflect". Perplexity-User is a different token and the wrong one to test: "Since a user requested the fetch, this fetcher generally ignores robots.txt rules."
Check it right now
Before changing anything, confirm what a crawler actually sees. The check is free, takes one URL and needs no account.
Or run the robots.txt tester against the same URL to see the exact rule that applies.
How to fix it
- Run the free AI crawler checker on the URL and read the robots.txt verdict for PerplexityBot.
- Fetch the live robots.txt and find the group that governs PerplexityBot: its own group if present, otherwise the star group.
- If the star group disallows the path, add a "User-agent: PerplexityBot" group with "Allow: /".
- Check the WAF. Perplexity documents a custom rule matching its user agents and its published IP ranges, with the action set to Allow and a priority above the blocking rules.
- Re-check with the tool, then allow up to 24 hours for Perplexity to reflect the change.
Why it happens again
A WAF rule and a robots.txt rule block the same bot from two different places, and each can be changed without the other. A bot-fight-mode change at the edge can refuse PerplexityBot while the robots.txt still reads allowed, which is why a robots.txt check alone does not always explain a missing result. Note also that Perplexity-User generally ignores robots.txt, so a user-initiated fetch still succeeding is not evidence that PerplexityBot is allowed. stillindexed monitors the robots.txt side of this per URL and raises ai_search_blocked, a warning, when a search bot that was allowed is newly disallowed.
stillindexed re-checks the URLs you give it every 30 minutes on Starter and alerts when a directive changes, at most 30 minutes after it does. It is a monitor rather than a crawler: it watches a list you choose and tells you when one of seven things changes. Start with a 14-day free trial, 3 sites, no card required, and a 30 day refund after that.
Catching it next time
Fixing it once is the easy half. The setting that caused this can be changed again by anyone with access, and the page will keep returning 200 while it happens.
An AI search crawler blocked from a site, on other platforms
Sources
Every claim about Perplexity above is from their own documentation, read on 2026-10-12. Platforms change their settings; if one of these is out of date, their page wins and we would like to know.