On 22 October 2026 we fetched the homepage and robots.txt of every domain in the official .gov registry, all 16,935 of them, using the list CISA publishes at cisagov/dotgov-data. 13,043 domains answered. 9,078 returned a 200 HTML homepage.
106 of those homepages are public content pages, from a town hall to a state legislature to a federal program, that tell search engines to stay out, and still did on a second fetch. 80 carry a noindex directive, 22 sit behind a robots.txt rule that blocks Googlebot from the homepage, and 4 do both. They are spread across 105 different final hosts. Every one is listed below by state, with the signal we saw.
The pipeline and the curation rules are the ones from our Tranco top-10,000 study, pointed at public bodies. Most of the 253 raw flags turned out to be parked domains, login portals and placeholder pages. They are counted separately and none of them is in the 106.
Headline numbers
| What we saw | Domains |
|---|---|
| Registered .gov domains attempted | 16,935 |
| Answered on at least one entry URL | 13,043 |
| Returned a 200 HTML homepage | 9,078 |
Raw homepage noindex or Googlebot robots.txt flags |
253 |
| Flagged again on a second fetch | 246 |
| Public content homepage telling Google to stay out, after hand review | 106 |
| Refused our request with 401, 403 or 429 (mostly by IP address, see section 4) | 3,585 |
| Certificate or TLS failure on at least one entry URL | 723 |
| Homepage ends on a host outside .gov | 553 |
robots.txt disallows GPTBot, ClaudeBot or CCBot from / |
741 |
No DNS answer for the domain or its www host |
2,153 |
The numbers, one line each
Each of these is a standalone finding, checked against the summary JSON, and can be quoted on its own.
106 of 9,078 reachable .gov homepages are public content pages that tell search engines not to index them, measured 22 October 2026, 12:56 to 15:20 UTC.
80 of those 106 carry a meta noindex tag, 22 sit behind a robots.txt rule that blocks Googlebot from the homepage, and 4 do both, measured 22 October 2026, 12:56 to 15:20 UTC.
148 of 9,078 .gov homepages that returned 200 HTML (1.6 percent) carried a robots meta tag with noindex, measured 22 October 2026, 12:56 to 14:13 UTC.
85 of 8,150 parseable robots.txt files (1.0 percent) disallowed Googlebot from the homepage, measured 22 October 2026, 12:56 to 14:13 UTC.
2,153 of 16,935 registered .gov domains (12.7 percent) had no DNS answer for the domain or its www host, measured 22 October 2026, 12:56 to 14:13 UTC.
3,585 of 16,935 registered .gov domains (21.2 percent) refused our request with 401, 403 or 429, and 87.6 percent of the 3,531 we probed again with a browser-shaped request refused that too, measured 22 October 2026.
723 of 16,935 registered .gov domains failed certificate validation or the TLS handshake on at least one entry URL, and 218 of them served nothing at all, measured 22 October 2026, 12:56 to 14:13 UTC.
741 of 8,150 parseable robots.txt files (9.1 percent) disallow GPTBot, ClaudeBot or CCBot from the root, measured 22 October 2026, 12:56 to 14:13 UTC.
1. Homepages that tell Google to stay out
Raw flags, before any judgement:
- 148 of 9,078 200 HTML homepages (1.6%) carried a robots
<meta>tag withnoindex. - 35 of 9,078 (0.4%) sent
noindexin anX-Robots-Tagresponse header. - 85 of 8,150 parseable robots.txt files (1.0%) disallowed Googlebot from
/or from the homepage path. The robots.txt file is the one on the host the homepage fetch ended on, which is not always the registered domain.
That is 253 domains. We fetched all of them again with the same binary between 14:50 and 15:20 UTC, and 246 were flagged again. A third request recorded what a person needs to judge each row: the page title, the first 300 characters of visible text, the robots meta line, the X-Robots-Tag header and the robots.txt groups for * and Googlebot. Then we sorted the 253 by hand, with the four classes from the Tranco study:
| Class | Domains | What it means |
|---|---|---|
| Apparent accident | 106 | A public content homepage, reproduced on the second fetch, that says noindex or sits behind a robots.txt rule blocking Googlebot. Nothing on the page marks it as a placeholder, a login, an archive or a redirect. |
| By design, or not a website | 103 | Parked domains (easyDNS, Zoneedit, registrar parking pages), hosting-provider default and error pages, under-construction and coming-soon pages, login and application portals (Everbridge alert sign-ups, Outlook Web App, password-protected WordPress, Cloudflare Access), archived sites such as bidenwhitehouse.gov, and redirect stubs. |
| Fetch artefact | 21 | Bot walls that answered our request with a noindex stub (Incapsula, Cloudflare), a geo-block page, the Federal Register’s scraper gate, and flags the second fetch did not reproduce. |
| Unclear | 23 | Could be either, so not counted. This includes the 13 TreasuryDirect domains (below), application storefronts, a homepage that is only the organisation’s name, a Googlebot-only block, and rows where the evidence fetch failed. |
“Apparent accident” is the Tranco study’s class name, kept so the two studies can be compared. It describes what the page shows, not what anybody intended. We have not contacted any of these bodies, and a site can have a reason it does not publish.
Google documents what each signal does. A noindex rule means Google “will drop that page entirely from Google Search results, regardless of whether other sites link to it” (Google Search Central). A robots.txt block stops crawling, not listing: “If your web page is blocked with a robots.txt file, its URL can still appear in search results, but the search result won’t have a description” (Google Search Central). So the 80 noindex homepages ask to be removed, and the 22 robots.txt homepages can still show up as a bare address with no description. On the 4 that do both, the robots.txt block means Google never reads the noindex: “If the page is blocked by a robots.txt file or the crawler can’t access the page, the crawler will never see the noindex rule” (Google Search Central).
TreasuryDirect, counted as unclear
treasurydirect.gov and the Treasury domains that land on the same site (savingsbonds.gov, treasuryhunt.gov, wizard.gov and others, 13 domains in all, each in the CSV) answer with this robots.txt:
User-agent: usasearch
Allow: /
Crawl-delay: 2
Sitemap: https://treasurydirect.gov/sitemap.xml
User-agent: *
Disallow: /
usasearch is the crawler of Search.gov, the government’s on-site search service, and digital.gov tells agencies to allow it (digital.gov). Digital.gov’s robots.txt guide shows the same shape, a usasearch allow plus a * disallow, scoped to an /archive/ folder, as the way to keep content in Search.gov but out of commercial search engines (digital.gov). From outside, the file cannot tell us whether that is the intent here, so these 13 are in the unclear row, not in the 106.
2. Who they are
13 of the 106 are federal executive branch domains and 1 is legislative. The rest are states, counties, cities, special districts, tribes and election offices.
- Arizona Legislature. azhouse.gov, azsenate.gov and azcapitolmuseum.gov each serve
<meta name='robots' content='noindex, nofollow' />on the homepage. That is the tag WordPress generates when “Discourage search engines from indexing this site” is ticked under Settings → Reading (WordPress documentation), and all three robots.txt files carry WordPress paths. Our WordPress noindex fix shows where the setting lives. - wa.gov and nv.gov publish
User-agent: *andDisallow: /. Both were among the 14 apparent accidents in the Tranco study in September and were still blocking at that study’s 1 October pass. - Federal. hudhomestore.gov (HUD’s listing of homes for sale), together.gov and juntos.gov (DHS), jimmycarterlibrary.gov (National Archives), fsd.gov (GSA’s Federal Service Desk) and ppdcecc.gov (the Congressional-Executive Commission on China’s political prisoner database) serve
User-agent: *withDisallow: /. The file on together.gov and juntos.gov opens with the comment# Block all robots from everything for dev and staging. stopfakes.gov and wwtg.gov (Commerce), tvaoig.gov, fedpartnership.gov, ibab.gov, casl.gov and usdebitcard.gov carry a homepagenoindex. collegenavigator.gov redirects tonces.ed.gov/collegenavigator/, a path the nces.ed.gov robots.txt disallows for all crawlers.
For the federal executive rows there is a published standard to read them against. OMB memorandum M-23-22, issued in September 2023 to implement the 21st Century IDEA, says agency websites must “follow search engine optimization (SEO) practices to ensure that members of the public can access government information and services from third-party websites and applications”, and that “the Federal Government’s public web presence is an open book that may be crawled, archived, or ‘scraped’ by anyone in the general public, at any time” (M-23-22; digital.gov still lists it as the current requirements, digital.gov). The memo applies to executive agencies, not to states, counties or cities. Whether a particular homepage fits it is for the agency to say. We publish what the homepage says.
By organization type
| Organization type (CISA) | Domains | Responded | 200 HTML homepage | Refused our request | Raw noindex | Raw robots block | Apparent accidents |
|---|---|---|---|---|---|---|---|
| City | 9,223 | 7,513 | 5,239 | 2,142 | 77 | 23 | 43 |
| County | 2,716 | 2,000 | 1,202 | 704 | 32 | 7 | 11 |
| State or territory | 1,438 | 883 | 580 | 237 | 19 | 19 | 11 |
| Federal - Executive | 1,174 | 859 | 664 | 167 | 12 | 21 | 13 |
| Special district | 1,113 | 846 | 690 | 130 | 14 | 4 | 13 |
| City - Election | 346 | 276 | 210 | 63 | 5 | 0 | 2 |
| County - Election | 325 | 263 | 164 | 82 | 3 | 3 | 3 |
| Tribal | 289 | 199 | 168 | 22 | 6 | 3 | 6 |
| Federal - Legislative | 121 | 86 | 69 | 16 | 0 | 3 | 1 |
| School district | 75 | 34 | 29 | 3 | 2 | 2 | 1 |
| State or territory - Election | 41 | 28 | 18 | 10 | 3 | 0 | 1 |
| Interstate | 39 | 33 | 24 | 7 | 2 | 0 | 1 |
| Federal - Judicial | 24 | 15 | 15 | 0 | 0 | 0 | 0 |
| Special district - Election | 5 | 3 | 2 | 1 | 0 | 0 | 0 |
| Tribal - Election | 4 | 3 | 3 | 0 | 2 | 0 | 0 |
| Federal | 2 | 2 | 1 | 1 | 0 | 0 | 0 |
“Raw” columns are first-pass flags before review. “Apparent accidents” is the reviewed count.
By state
| State | Domains | Responded | 200 HTML homepage | Refused our request | Raw noindex | Raw robots block | Apparent accidents |
|---|---|---|---|---|---|---|---|
| (blank) | 3 | 3 | 2 | 1 | 0 | 0 | 0 |
| AK | 66 | 46 | 33 | 11 | 4 | 1 | 1 |
| AL | 272 | 194 | 140 | 44 | 7 | 1 | 5 |
| AR | 415 | 234 | 191 | 39 | 7 | 1 | 0 |
| AS | 6 | 5 | 5 | 0 | 0 | 0 | 0 |
| AZ | 292 | 226 | 103 | 116 | 3 | 0 | 3 |
| CA | 893 | 682 | 434 | 227 | 7 | 6 | 4 |
| CO | 295 | 225 | 149 | 74 | 2 | 1 | 1 |
| CT | 191 | 161 | 80 | 75 | 0 | 0 | 0 |
| DC | 990 | 719 | 552 | 148 | 8 | 23 | 12 |
| DE | 24 | 23 | 17 | 6 | 0 | 0 | 0 |
| FL | 555 | 430 | 276 | 143 | 6 | 1 | 2 |
| GA | 375 | 307 | 207 | 92 | 3 | 3 | 2 |
| GU | 3 | 3 | 2 | 0 | 0 | 0 | 0 |
| HI | 16 | 11 | 8 | 3 | 1 | 0 | 0 |
| IA | 233 | 181 | 126 | 49 | 3 | 1 | 1 |
| ID | 117 | 84 | 65 | 18 | 2 | 1 | 1 |
| IL | 470 | 356 | 252 | 97 | 8 | 3 | 4 |
| IN | 156 | 98 | 71 | 23 | 0 | 0 | 0 |
| KS | 257 | 190 | 122 | 64 | 4 | 1 | 4 |
| KY | 241 | 187 | 118 | 60 | 1 | 1 | 1 |
| LA | 132 | 90 | 66 | 18 | 1 | 0 | 1 |
| MA | 376 | 323 | 155 | 165 | 0 | 2 | 1 |
| MD | 278 | 229 | 165 | 56 | 3 | 2 | 2 |
| ME | 151 | 130 | 84 | 43 | 1 | 1 | 0 |
| MI | 656 | 531 | 390 | 134 | 6 | 0 | 5 |
| MN | 631 | 543 | 375 | 159 | 2 | 2 | 2 |
| MO | 376 | 268 | 204 | 59 | 6 | 2 | 4 |
| MP | 3 | 2 | 2 | 0 | 0 | 0 | 0 |
| MS | 135 | 80 | 64 | 14 | 2 | 0 | 2 |
| MT | 144 | 108 | 77 | 28 | 0 | 1 | 0 |
| NC | 417 | 320 | 204 | 102 | 3 | 1 | 2 |
| ND | 78 | 54 | 41 | 11 | 2 | 1 | 1 |
| NE | 163 | 114 | 64 | 15 | 2 | 0 | 2 |
| NH | 235 | 190 | 110 | 75 | 0 | 0 | 0 |
| NJ | 337 | 261 | 179 | 74 | 10 | 3 | 2 |
| NM | 111 | 95 | 63 | 29 | 1 | 1 | 1 |
| NV | 57 | 43 | 28 | 15 | 4 | 3 | 3 |
| NY | 1,447 | 1,246 | 964 | 265 | 12 | 3 | 4 |
| OH | 706 | 503 | 358 | 115 | 9 | 2 | 3 |
| OK | 278 | 167 | 140 | 20 | 0 | 0 | 0 |
| OR | 299 | 229 | 156 | 64 | 2 | 2 | 2 |
| PA | 397 | 290 | 220 | 58 | 7 | 4 | 6 |
| PR | 8 | 5 | 4 | 0 | 0 | 0 | 0 |
| RI | 64 | 56 | 28 | 26 | 2 | 0 | 0 |
| SC | 157 | 120 | 76 | 36 | 2 | 1 | 2 |
| SD | 73 | 55 | 46 | 9 | 2 | 0 | 2 |
| TN | 375 | 290 | 199 | 82 | 4 | 2 | 3 |
| TX | 765 | 618 | 402 | 194 | 4 | 4 | 0 |
| UT | 407 | 351 | 277 | 72 | 5 | 1 | 4 |
| VA | 346 | 242 | 148 | 89 | 1 | 0 | 0 |
| VI | 6 | 6 | 6 | 0 | 1 | 0 | 0 |
| VT | 104 | 92 | 66 | 24 | 1 | 0 | 0 |
| WA | 264 | 206 | 114 | 88 | 4 | 1 | 3 |
| WI | 843 | 656 | 528 | 121 | 10 | 2 | 6 |
| WV | 164 | 101 | 83 | 16 | 1 | 0 | 1 |
| WY | 82 | 64 | 39 | 19 | 1 | 0 | 1 |
All 106, by state
Each row was flagged on the first fetch, flagged again on the second, and reviewed by hand against the third. An arrow shows where the homepage fetch ended when that was a different host; the signal belongs to that host.
| State | Domain | Organization | Type | Signal on both fetches | Note |
|---|---|---|---|---|---|
| AK | ccthita-nsn.gov → tlingithaida.kinsta.cloud |
Central Council Tlingit and Haida Indian Tribes of Alaska | Tribal | X-Robots-Tag noindex | redirects to a staging host |
| AL | alsar.gov | Alabama Search and Rescue | Special district | meta noindex + X-Robots-Tag noindex | |
| AL | dekalbcountyal911.gov | DeKalb County E-911 | Special district | meta noindex | |
| AL | hancevillewateral.gov | Hanceville Water & Sewer Board | Special district | meta noindex | |
| AL | pinsonal.gov | City of Pinson, Al. | City | meta noindex | |
| AL | sardiscityal.gov | Sardis City Alabama | City | meta noindex | |
| AZ | azcapitolmuseum.gov | Arizona Legislative Council | State or territory | meta noindex | |
| AZ | azhouse.gov | Arizona House of Representatives | State or territory | meta noindex | |
| AZ | azsenate.gov | Arizona Senate | State or territory | meta noindex | |
| CA | brb-nsn.gov → www.bearriverrancheria.org |
BEAR RIVER BAND OF THE ROHNERVILLE RANCHERIA | Tribal | meta noindex | |
| CA | joinnovatopolice.gov | City of Novato | City | X-Robots-Tag noindex | |
| CA | shastacoeducation.gov → www.shastacoe.org |
Shasta County Office of Education | School district | meta noindex + robots.txt blocks Googlebot | |
| CA | ssvemsca.gov | Sierra-Sacramento Valley Emergency Medical Services Agency | Special district | meta noindex | |
| CO | hireacoloradovet.gov | Colorado Dept Labor and Employment | State or territory | meta noindex + robots.txt blocks Googlebot | |
| DC | collegenavigator.gov → nces.ed.gov |
Department of Education (Office of the Chief Information Officer) | Federal - Executive | robots.txt blocks Googlebot | nces.ed.gov robots.txt disallows /collegenavigator/ |
| DC | fedpartnership.gov | Board of Governors of the Federal Reserve | Federal - Executive | meta noindex | |
| DC | fsd.gov | General Services Administration (Office of GSA IT) | Federal - Executive | robots.txt blocks Googlebot | |
| DC | hudhomestore.gov | Department of Housing and Urban Development | Federal - Executive | robots.txt blocks Googlebot | |
| DC | ibab.gov | United States Agency for Global Media | Federal - Executive | meta noindex | |
| DC | juntos.gov | Department of Homeland Security (Management Directorate) | Federal - Executive | robots.txt blocks Googlebot | |
| DC | ppdcecc.gov | United States House of Representatives (Congressional-Executive Commission on China) | Federal - Legislative | robots.txt blocks Googlebot | |
| DC | stopfakes.gov | Department of Commerce (International Trade Administration) | Federal - Executive | meta noindex | |
| DC | together.gov | Department of Homeland Security (Management Directorate) | Federal - Executive | robots.txt blocks Googlebot | |
| DC | usdebitcard.gov | Department of the Treasury (Bureau of the Fiscal Service) | Federal - Executive | meta noindex | public program homepage with cardholder login links |
| DC | wmsc.gov | Washington Metrorail Safety Commission | Interstate | meta noindex | |
| DC | wwtg.gov | Department of Commerce (International Trade Administration) | Federal - Executive | meta noindex | |
| FL | cityofcenterhill.gov | City of Center Hill | City | meta noindex | |
| FL | jeffersoncountyfl.gov | Jefferson County Board of County Commissioners | County | meta noindex | |
| GA | cedartowngeorgia.gov | City of Cedartown | City | robots.txt blocks Googlebot | |
| GA | lincolncountyga.gov | Lincoln County Government | County | meta noindex + robots.txt blocks Googlebot | |
| IA | bvcountyattorney.gov | Buena Vista County | County | meta noindex | |
| ID | nampaauditoriumdistrict.gov | Nampa Auditorium District | Special district | meta noindex | |
| IL | menardcountyil.gov | Menard County, Illinois | County | meta noindex | |
| IL | platotwp.gov | Plato Township | City | meta noindex | |
| IL | rocktontownshipil.gov | Rockton Township | City | meta noindex | |
| IL | unioncountyil.gov | Union County | County | meta noindex | |
| KS | fontanaks.gov | City of Fontana, Kansas | City | meta noindex | |
| KS | galenaks.gov | City of Galena | City | meta noindex | |
| KS | kalapks.gov | Kansas Judicial Branch | State or territory | meta noindex | |
| KS | linnvalley.gov | City Of Linn Valley | City | meta noindex | |
| KY | fayettekyclerk.gov | Fayette County Clerk | County - Election | robots.txt blocks Googlebot | |
| LA | townofmaurice.gov | Town Of Maurice | City | meta noindex | |
| MA | masscomparecare.gov | Center for Health Information and Analysis | State or territory | robots.txt blocks Googlebot | |
| MD | carrollcountymd.gov | Commissioners of Carroll County, Maryland | County | robots.txt blocks Googlebot | |
| MD | jimmycarterlibrary.gov | National Archives and Records Administration | Federal - Executive | robots.txt blocks Googlebot | |
| MI | resorttownship.gov | Resort Township | City - Election | meta noindex | |
| MI | rollintownship.gov | Rollin Township | City - Election | meta noindex | |
| MI | somersettownshipmi.gov | Somerset Township | City | meta noindex | |
| MI | stjosephcountymi.gov | St. Joseph County Michigan Government | County - Election | meta noindex | |
| MI | villageofromeomi.gov | Village of Romeo | City | meta noindex | |
| MN | collegevilletownship.gov | Collegeville Township | City | meta noindex | |
| MN | millelacsband-nsn.gov | Mille Lacs Band of Ojibwe | Tribal | robots.txt blocks Googlebot | |
| MO | howellcountysheriff.gov | Howell County Sheriff’s Office | County | meta noindex | |
| MO | nemo911.gov | Marion County Emergency Services | Special district | meta noindex | |
| MO | portagedessiouxmo.gov | City of Portage Des Sioux, MO | City | meta noindex | |
| MO | viburnum.gov → www.cityofviburnum.com |
City of Viburnum | City | meta noindex | |
| MS | cantonms.gov | City of Canton, MS | City | meta noindex | |
| MS | learnedms.gov | Town of Learned | City | meta noindex | |
| NC | riegelwoodsanitary.gov → www.riegelwoodnc.com |
Riegelwood Sanitary District | Special district | meta noindex | |
| NC | rockinghampd.gov | Rockingham Police Department | City | meta noindex | |
| ND | bottineauambulance.gov | Bottineau Ambulance Service District | Special district | meta noindex + robots.txt blocks Googlebot | |
| NE | cityofhendersonne.gov | City of Henderson | City | meta noindex | |
| NE | districtclerkcheyennecountyne.gov | Clerk of the District Court | County | meta noindex | |
| NJ | evesham-nj.gov | Evesham Township | City | meta noindex | |
| NJ | frenchtownboro.gov | Frenchtown Borough | City | meta noindex | |
| NM | nmjusticedata.gov | NM Administrative Office of the Courts | State or territory | robots.txt blocks Googlebot | |
| NV | nv.gov | Governor’s Technology Office, State of Nevada | State or territory | robots.txt blocks Googlebot | |
| NV | nvggms.gov → www.nevadatreasurer.nv.gov |
Nevada State Treasurer | State or territory | robots.txt blocks Googlebot | |
| NV | nvprepaid.gov → www.nevadatreasurer.nv.gov |
Nevada State Treasurer | State or territory | robots.txt blocks Googlebot | |
| NY | brewertonfiredistrict.gov | Brewerton Fire District | Special district | meta noindex | |
| NY | gallatinny.gov | Town of Gallatin | City | robots.txt blocks Googlebot | |
| NY | senecacountyhealthny.gov | Seneca County Health Department | County | meta noindex | |
| NY | townofwheatlandny.gov | Town of Wheatland NY | City | meta noindex | |
| OH | brunswickohio.gov | City of Brunswick | City | meta noindex | |
| OH | loraincommonpleascourt.gov | Lorain County Court of Common Pleas | County | robots.txt blocks Googlebot | |
| OH | warrentwptrumbull.gov | Warren Township | City | meta noindex | |
| OR | cityofsodaville.gov | Sodaville | City | meta noindex | |
| OR | roseburgor.gov | City of Roseburg | City | robots.txt blocks Googlebot | |
| PA | berwickpa.gov | Borough of Berwick | City | meta noindex | |
| PA | betheltwplebanon.gov | Bethel Township | City | meta noindex | |
| PA | cantonboroughpa.gov | Canton Borough | City | meta noindex | |
| PA | hamptonshalerwater.gov → hswa-pa.org |
Hampton Shaler Water Authority | Special district | meta noindex | |
| PA | westhomesteadpa.gov → westhomesteadpa.com |
West Homestead Borough | City | robots.txt blocks Googlebot | |
| PA | westyorkpa.gov | West York Borough | City | meta noindex | |
| SC | corinth-shilohfd.gov | Corinth-Shiloh Fire Tax District | Special district | meta noindex | |
| SC | workscelections.gov | SC State Election Commission | State or territory - Election | meta noindex | |
| SD | brulecounty.gov | Brule County | County - Election | meta noindex | |
| SD | wakpamnilake-nsn.gov | Wakpamni Lake Community | Tribal | meta noindex | |
| TN | auburntown.gov | City of Auburntown | City | meta noindex | |
| TN | casl.gov | Department of Energy (Oak Ridge National Laboratory) | Federal - Executive | meta noindex | |
| TN | tvaoig.gov | Tennessee Valley Authority | Federal - Executive | meta noindex | |
| UT | bicknellutah.gov | Bicknell Town | City | meta noindex | |
| UT | bigwaterut.gov | Big Water Municipal | City | meta noindex | |
| UT | lccwaterutah.gov | Salt Lake County Service area #3 | Special district | meta noindex | |
| UT | transjordanutah.gov | Trans-Jordan Cities Agency | Special district | meta noindex | |
| WA | townofwilkeson.gov → www.townofwilkeson.com |
Town of Wilkeson | City | meta noindex | |
| WA | tulaliptribalcourt-nsn.gov | Tulalip Tribes of Washington | Tribal | meta noindex | |
| WA | wa.gov | State of Washington - Consolidated Technology Services | State or territory | robots.txt blocks Googlebot | |
| WI | buenavistarcwi.gov | Town of Buena Vista | City | meta noindex | |
| WI | mohican-nsn.gov | Stockbridge-Munsee Band of Mohican Indians | Tribal | meta noindex | |
| WI | townofakan.gov | Town of Akan | City | meta noindex | |
| WI | townofbalsamlakewi.gov | Town of Balsam Lake | City | meta noindex | |
| WI | townofhenriettawi.gov → henrietta.my-staging-website.com |
Henrietta Township | City | meta noindex | redirects to a staging host |
| WI | townofrudolphwi.gov | Town of Rudolph | City | robots.txt blocks Googlebot | |
| WV | masoncountyoeswv.gov | Mason County Emergency Services | County | meta noindex | |
| WY | cokevillewy.gov | Town of Cokeville | City | meta noindex |
3. Errors that are not about Google
- No DNS. 2,153 domains (12.7%) had no DNS answer for the domain or its
wwwhost. CISA’s own README says “not all registered domains offer an online service (e.g., a website, an email server) at the domain” (cisagov/dotgov-data), so we do not call these broken. - No response. 1,521 more (9.0%) failed on all three entry URLs for other reasons, such as timeouts, refused connections, resets and redirect loops.
- HTTP errors. 208 homepages answered with a 4xx other than 401, 403 or 429, 150 of them 404 or 410, and 202 did the same on the second fetch. 147 answered with a 5xx, 129 of them again on the second fetch.
4. 3,585 refusals, and why they are about us
3,585 domains (21.2%) answered our homepage request with 401, 403 or 429, 3,544 of them with 403. 3,516 refused again on the second pass.
This study ran from one residential connection in Istanbul. To separate “this server refuses bots” from “this server refuses this address”, we sent 3,531 of the refused homepages one browser-shaped request, with the same headers the product uses for its own bot-wall check. 3,093 of 3,531 (87.6%) refused the browser-shaped request too, 422 served it a 200, and 16 answered something else. Most of these refusals key on where the request comes from, not on what it calls itself.
Google says “the default IP addresses of the Googlebot crawler appear to be based in the USA” (Google Search Central). A server that refuses non-US traffic is not, by that fact, refusing Googlebot. None of the 3,585 is counted as a finding, and none is in the 106. It also means the 106 come from the 9,078 homepages we could read: the refused domains could hold more.
5. Certificates
723 domains failed certificate validation or the TLS handshake on at least one of the three entry URLs. 685 failed the same way on the second fetch.
| Failure | Domains |
|---|---|
| Certificate does not cover the name | 450 |
| TLS handshake failure | 147 |
| Certificate chain not trusted | 73 |
| Rejected by the macOS verifier as not standards compliant | 26 |
| Certificate expired | 24 |
| Other TLS error | 3 |
505 of the 723 still served a homepage through another entry URL (www or plain http). 218 served nothing at all. The 26 macOS rejections are a property of the client: this crawl ran on macOS, whose verifier enforces Apple’s certificate policy, and another client may accept the same certificate.
This matters more on .gov than elsewhere. CISA preloads every newly registered .gov domain into the browser HSTS list, which removes “the ability to click through a certificate error” (get.gov). On a preloaded domain, a visitor who meets one of these certificates cannot continue to the site. We did not check which of the 723 are preloaded.
6. Homepages that leave .gov
553 homepages (3.3%) ended on a host outside .gov after redirects: 246 on .com, 211 on .org, 53 on .us and 29 on .net, with the rest spread over other endings. 525 did the same on the second fetch. 8 are federal domains.
Many small governments run their main site on an older .org or .us address and point the .gov at it. That is a choice, not an error. It does mean a resident who follows the .gov link ends on an address that does not say government, which is the thing the .gov domain exists to say: the registry describes its purpose as making “the online services of U.S.-based government organizations … easy to identify on the internet” (cisagov/dotgov-data).
7. AI crawlers
Secondary, and reported as raw counts without review. Of 8,150 parseable robots.txt files:
- 741 (9.1%) disallow at least one of GPTBot, ClaudeBot or CCBot from
/, by name or through aUser-agent: *rule that covers them. - GPTBot is disallowed in 721 (8.8%), ClaudeBot in 699 (8.6%), CCBot in 479 (5.9%).
- Among federal domains, 50 of 663 parseable files (7.5%) disallow at least one of the three.
M-23-22 mentions this use directly: scraping of federal websites matters “potentially for the training of large language models that enable artificial intelligence chatbots and services to accurately represent information about the government.”
A line that reads differently depending on the parser. 44 of the 7,841 robots.txt files we fetched again contain DisallowAITraining: / inside the User-agent: * group. Google’s documentation lists the fields it supports as user-agent, allow, disallow and sitemap, and says other fields “aren’t supported” (Google Search Central). Google’s open-source robots.txt parser matches field names by prefix: KeyIsDisallow accepts any key that starts with “disallow” (google/robotstxt, robots.cc). Under that parser, all 44 files read as Disallow: / for Googlebot. Under the parser this study used, none of them blocks Googlebot, and none is among the 85 robots flags. Which reading Google’s production crawler applies is not documented, so we report the 44 as a count and not as a block.
What this found in our own tools
Two problems in our own pipeline surfaced during the run. Both are fixed, and both are recorded here because a monitor that hides its own false readings is not one to trust with yours.
- A bot-wall test that fired on contact forms. The survey marked any 200 page that referenced
www.google.com/recaptchaas a challenge page. On .gov that includes sites that load reCAPTCHA for a contact form, and it put 812 homepages that answered 200 into the bot-wall bucket. The test now looks for a canonical that points at reCAPTCHA, with a regression test. This study counts those 812 as the 200 responses they were. - A parser change in the middle of the study. The first build of the second pass picked up a new robots.txt evaluator that read
DisallowAITraining: /as a Googlebot disallow, so the second pass flagged robots blocks the first pass had not. We discarded that pass and ran it again from the same commit as the first. The evaluator now ignores fields that only start with “disallow”, with tests, and theDisallowAITrainingcount above comes from that investigation.
Method
Source. current-full.csv from cisagov/dotgov-data at commit 718c7ac8deb13ee4917d45101ee6a0dd08fb94f2 (committed 2026-10-22T11:23:57Z): 16,935 registered second-level domains with CISA’s domain type, organization and state. The security-contact column is not republished.
First pass. The survey command from the stillindexed repository, built at commit 8edd154, the same tool behind the Tranco study. User agent stillindexed-survey/1 (+https://stillindexed.com/bot), concurrency 24, at most one in-flight request per host, 10 second timeout, no retries, 2 MiB body cap, 10 redirect hops. Entry order https://<domain>/, then https://www.<domain>/, then http://<domain>/; robots.txt from the host the homepage ended on; first declared sitemap; TLS facts from the same connection. Ran from 12:56 to 14:13 UTC on 22 October 2026. Median 2,657 ms per responding domain, p90 21,134 ms.
Second pass. The same binary over every domain the first pass flagged for a noindex, robots.txt, AI-crawler, HTTP error, certificate or off-.gov finding: 5,722 domains, from 14:50 to 15:20 UTC.
Evidence and probes. One GET per flagged homepage and its robots.txt for the hand review, one browser-shaped GET per refused homepage, and one GET per robots.txt that answered 200 on the first pass for the DisallowAITraining count. Each used a 10 second timeout and at most 8 requests at a time.
Parsers. The product’s own: robots meta and X-Robots-Tag parsing, robots.txt evaluation per RFC 9309 with Google’s group rules and exact product-token matching, the same redirect-chain recording as the monitor. A challenge page is a 403 or 503 carrying cf-mitigated: challenge, a cf-chl marker, or a “Just a moment” or “Attention Required” title.
Curation rules. A flag is an apparent accident only if all of these hold: the flag appears on both survey passes; the evidence fetch shows a public content homepage with navigation and content; nothing marks it as a placeholder, parking page, login, application shell, archive or redirect stub; and the robots.txt case is a rule that blocks Googlebot from the homepage, not a Googlebot-only group, which we count as unclear. Every row’s class and a one-line note are in the CSV.
Limitations
- One vantage point. A residential connection in Istanbul, Turkey. US government sites that refuse non-US traffic refused us; see section 4.
- HTTP only. No JavaScript. A homepage that adds or removes
noindexin the browser is recorded as its HTML says. - Homepage and robots.txt only. A site whose homepage is fine can still hide every other page.
- One afternoon. Sites change. The CSV records what was served on 22 October 2026.
- Registered domains, not websites. The registry lists every second-level .gov domain, including ones that never hosted a site.
- Human judgement. The review is ours. The CSV carries the class and a note for every reviewed row, so a reader can disagree row by row.
- Second-pass-only flags. 6 domains were flagged only on the second pass. They are not reviewed or counted.
Prior work
Singanamalla and colleagues measured HTTPS adoption and certificate errors across 135,408 government websites worldwide (IMC ’20). Sun, Zhuang and Giles studied robots.txt usage across 7,593 sites including government sites (WWW 2007). In the work we reviewed, we did not find a study that classifies .gov homepage noindex and robots.txt blocks by hand, separates them from parked and placeholder domains, and names the public content homepages.
Cite this dataset
CC BY 4.0. The machine-readable version of everything on this page is at
/research/homepage-indexability-dotgov-2026/citations.json.
APA
Stillindexed. (2026, October 22). Homepage indexability of every registered .gov domain: 106 public homepages that tell Google to stay out [Data set]. https://stillindexed.com/research/homepage-indexability-dotgov-2026
BibTeX
@dataset{stillindexed_dotgov_indexability_2026,
author = {{Stillindexed}},
title = {Homepage indexability of every registered .gov domain:
106 public homepages that tell Google to stay out},
year = {2026},
month = {10},
day = {22},
url = {https://stillindexed.com/research/homepage-indexability-dotgov-2026},
note = {Crawl of all 16,935 registered .gov domains from
cisagov/dotgov-data, commit 718c7ac8deb13ee4917d45101ee6a0dd08fb94f2},
license = {CC-BY-4.0}
}
Plain text
Stillindexed, homepage indexability of every registered .gov domain, 22 October 2026, https://stillindexed.com/research/homepage-indexability-dotgov-2026 (CC BY 4.0).
Reproduce
Download the CSV: one row per registered domain with CISA’s metadata, the outcome of both passes, every flag, the browser-probe status and the curation class and note. The aggregate figures are in the summary JSON.
The pipeline is in the stillindexed repository, which is not public yet:
node scripts/research/dotgov-input.mjs --ref 718c7ac8deb13ee4917d45101ee6a0dd08fb94f2
git worktree add --detach /tmp/survey-8edd154 8edd154
(cd /tmp/survey-8edd154/backend && go build -o /tmp/survey ./cmd/survey)
date -u > scratchpad/research/dotgov-2026/run-started.txt
/tmp/survey -in scratchpad/research/dotgov-2026/domains.txt \
-out scratchpad/research/dotgov-2026/results.jsonl -concurrency 24 -timeout 10s
date -u > scratchpad/research/dotgov-2026/run-finished.txt
node scripts/research/dotgov-recheck.mjs
node scripts/research/dotgov-robots-fields.mjs
node scripts/research/dotgov-aggregate.mjs
node scripts/research/dotgov-check.mjs
Any single row can be checked with curl. The Arizona Senate homepage:
curl -s -A 'stillindexed-survey/1 (+https://stillindexed.com/bot)' https://www.azsenate.gov/ \
| grep -io '<meta[^>]*robots[^>]*>'
Check your own
If you run a public site, the indexability checker reads your homepage the same way this study did, and the robots.txt tester shows what each crawler is allowed to fetch. For a page that has already dropped out of Google, deindex triage walks through the likely causes in order. The platform fixes for the two most common shapes in this list are WordPress noindex and Wix noindex.
If you would rather hear about the change than find it later, stillindexed watches robots.txt and homepage directives and alerts when they change. Details are on the robots.txt monitoring page.