Google-Extended: what it blocks and what it does not
Google-Extended is a control token, not a crawler. Blocking it stops Gemini training and grounding, not Google Search or AI Overviews.
Google-Extended is a standalone robots.txt token that controls whether Google may use crawled content to train Gemini models and to ground Gemini answers. It is not a crawler, and blocking it does not affect Google Search or AI Overviews, which follow Googlebot. Block it to refuse Gemini use while staying in Search.
Why Google does this
Google's own documentation is unambiguous, and this is the most commonly misread token in the list. Google-Extended is a token for "training future generations of Gemini models ... and for grounding (providing content from the Google Search index to the model at prompt time)", and "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search." It is not a separate crawler: "Google-Extended doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity." Google Search and its features, including AI Overviews, are served by Googlebot and follow the Googlebot rules, not this token. The misconception runs both ways. Blocking Google-Extended cannot remove a site from Search, so it is not the cause of a traffic drop. Allowing it does not put a site into Gemini grounding either, because that is a separate consent from being crawled.
Check it right now
Before changing anything, confirm what a crawler actually sees. The check is free, takes one URL and needs no account.
Or run the robots.txt tester against the same URL to see the exact rule that applies.
How to fix it
- Run the free AI crawler checker on the URL and read the status for "Google-Extended". It is a control token with no user agent, so the robots.txt rule is the only evidence and that is what the check reads.
- If you want Google out of Gemini training and grounding while staying in Search, keep the "User-agent: Google-Extended" Disallow. It does not touch Googlebot.
- If a page is missing from Google Search, Google-Extended is not the cause. Check the Googlebot rules, the page-level noindex and the status code with the free indexability checker.
- If you want Google in Search but out of Gemini, allow Googlebot and disallow Google-Extended. Those two are the supported combination.
- Re-check with the tool and re-read the live robots.txt to confirm both rules read the way you intend.
Why it happens again
A "block AI crawlers" preset usually turns on every AI token at once, so Google-Extended can be added next to the Googlebot rules by a plugin or a theme rather than by a person who read what each token does. The two sets are edited in different places, and one can change without the other. stillindexed watches the rules for the AI tokens on each monitored URL and raises ai_search_blocked, a warning, when a search or grounding bot that was allowed is newly disallowed. A training token change is recorded as ai_training_blocked at info and never alerts.
stillindexed re-checks the URLs you give it every 30 minutes on Starter and alerts when a directive changes, at most 30 minutes after it does. It is a monitor rather than a crawler: it watches a list you choose and tells you when one of seven things changes. Start with a 14-day free trial, 3 sites, no card required, and a 30 day refund after that.
Catching it next time
Fixing it once is the easy half. The setting that caused this can be changed again by anyone with access, and the page will keep returning 200 while it happens.
An AI search crawler blocked from a site, on other platforms
Sources
Every claim about Google above is from their own documentation, read on 2026-10-12. Platforms change their settings; if one of these is out of date, their page wins and we would like to know.