Crawler versus browser
Fetches one URL four times, as Googlebot, Bingbot, GPTBot and an ordinary browser, then compares what came back. It exists because a bot wall can refuse a crawler while serving you the page perfectly, and nothing in your own browser will ever show you that.
Four fetches, one per crawler, sent one at a time with a pause between them. That is why this takes a few seconds and costs four of your 60 hourly checks.
Illustrative result for a reserved example domain. Your check replaces it.
1 crawler is being turned away
A bot wall refused them while serving a browser normally. Your pages may not be getting crawled.
| Client | Status | Title | Canonical | Difference |
|---|---|---|---|---|
| Browserbaseline | 200 | Pricing | https://example.com/pricing | what a visitor sees |
| Googlebot | 403bot wall | none | none |
|
| Bingbot | 200 | Pricing | https://example.com/pricing | same as browser |
| GPTBot | 200 | Pricing | https://example.com/pricing | same as browser |
User agent strings are simulated from our network. IP allowlists may give verified crawlers a different response.
What this check means
- Bot wall
- A bot wall is an edge product that blocks traffic it judges automated, before the request reaches the site.
- Cloaking
- Cloaking is deliberately serving search engines different content from human visitors, and it is against Google spam policy.
- User agent
- A user agent is the string a client sends to identify itself. It is trivially forged, so it identifies intent rather than proving identity.
- Baseline
- The baseline is the browser response. Every crawler result is compared against it, so if the browser is refused too there is nothing to compare.
How to read the result
- Status difference
- A crawler receiving a different HTTP status from a browser. This is the clearest signal in the check and usually means an edge rule.
- Bot wall
- The response carried a bot-management fingerprint, or a browser-shaped request got past where the crawler did not. Reported as us being refused, not as your page being broken.
- Indexability directives
- A different noindex, canonical or X-Robots-Tag per crawler. Rare, serious, and almost always a misconfigured edge rule rather than intent.
- Content difference
- Same status, different bytes. Reported at low confidence on purpose: a timestamp or an experiment causes this as readily as cloaking, and calling it cloaking would be an accusation.
Common causes
- Cloudflare Bot Fight Mode, which blocks unverified crawlers by default.
- A WAF rule written to stop scrapers that catches search engines too.
- Rate limiting that trips on crawl bursts and returns 429.
- Edge middleware serving a different variant based on user agent.
- A CDN cache serving a stale or error page to non-browser clients.
Why would Googlebot get a different page than I do?
Usually a bot wall. Cloudflare Bot Fight Mode, DataDome, PerimeterX and similar products refuse traffic they judge automated, and a crawler is automated by definition. The site owner never sees it, because in their own browser the page loads perfectly.
Does this prove cloaking?
No, and it deliberately does not claim to. A difference in status or in indexability directives is strong evidence. A difference in page content alone is reported at low confidence, because a timestamp, a CSRF token or an A/B test produces one just as easily as cloaking does.
Is this the same as verified Googlebot?
No. We send Googlebot's user agent from our own network. A site that allowlists Google by IP or verifies by reverse DNS may treat the real Googlebot differently from this check. The result tells you how the site treats something that looks like a crawler.
What should I do if a crawler is being blocked?
Allowlist it at your edge. In Cloudflare that means a WAF skip rule for verified bots, or turning off Bot Fight Mode for the paths that need indexing. Verify the crawler by reverse DNS rather than by user agent, since a user agent is trivially forged.
Why does this take longer than the other tools?
It makes four requests rather than one, and sends them one at a time with a pause between them. Firing four at once would be faster and would also be exactly the behaviour that gets a crawler blocked in the first place.