Sitemap hygiene check
A sitemap is a list of the URLs you want Google to index. This audit fetches up to 200 of them and flags every one that contradicts that: answers something other than 200, redirects, says noindex in a meta tag or header, is blocked by robots.txt for Googlebot, or names another URL as canonical. Enter a sitemap URL, or a domain to use the sitemap its robots.txt declares. The result exports as CSV.
Up to 200 URLs, fetched at no more than about three a second with our published stillindexedbot user agent. A full audit takes up to 90 seconds. Two audits an hour per IP, separate from the 60-fetch budget the other tools share.
Illustrative result for a reserved example domain. Your audit replaces it.
5 of 7 URLs need attention
A sitemap should list only the URLs you want in Google, at the address Google should index. The flagged rows below are not that.
- Sitemap
- https://example.com/sitemap.xml (declared in robots.txt)
- Listed URLs read
- 7
- Sample
- all 7
- lastmod
- 7 of 7 URLs carry one; every value is identical (2026-10-22), so Google cannot use it to tell what changed
| URL | Status | Flags | Detail |
|---|---|---|---|
| https://example.com/old-plans | 200 | redirected | final URL https://example.com/pricing |
| https://example.com/pricing?ref=nav | 200 | canonical elsewhere | canonical https://example.com/pricing |
| https://example.com/blog/draft | 200 | noindex | meta robots: noindex |
| https://example.com/team/ana | 404 | not 200 | — |
| https://example.com/search?q=shoes | 200 | blocked by robots.txt | — |
Sitemaps read (1)
- https://example.com/sitemap.xml: HTTP 200, urlset, 7 entries
Fetched as stillindexedbot, not as Googlebot. A URL that robots.txt disallows for crawlers in the * group is not fetched at all, so it is marked not checked rather than guessed at.
A sitemap is regenerated on every deploy, so a clean audit today says nothing about the next build. stillindexed watches the sitemap and alerts when it stops serving, stops parsing or loses more than half its URLs. The single-file check is the sitemap validator.
Want this checked every 30 minutes?
14-day free trial, 3 sites, no card required. We re-check the URLs you choose, confirm a critical change with a second fetch, then alert Telegram and email.
What this check means
- Sitemap hygiene
- A sitemap is clean when every URL it lists is one you want indexed, at the address Google should index: it answers 200, is crawlable and indexable, and names itself as canonical.
- Sample
- Up to 200 URLs drawn evenly from everything read. Smaller sitemaps are audited in full.
- Googlebot verdict
- Each URL is evaluated against its own host’s robots.txt for Googlebot, the same evaluation the robots.txt tester runs. A URL on a host whose robots.txt could not be read is marked unknown.
- Not checked
- A URL we did not fetch: robots.txt disallows our crawler, the site asked us to slow down, or the audit reached 90 seconds. It is never counted as clean.
How to read the result
- not 200
- The final answer was not 200: a 404 or 410 for a removed page, a 5xx, or a 403 from the edge. Google does not index URLs that answer 4xx.
- redirected
- The listed URL redirects. List the target instead; the redirect tells Google the target is the one to index.
- noindex
- The page carries noindex in a meta robots or meta googlebot tag, or in the X-Robots-Tag header, including directives scoped to googlebot. The detail names which.
- blocked by robots.txt
- robots.txt disallows Googlebot from the URL, so Google cannot crawl what the sitemap asks it to.
- canonical elsewhere
- The page names a different URL in rel="canonical". Either the sitemap lists a duplicate, or the canonical is wrong.
- lastmod
- Whether lastmod is present, identical on every URL, dated in the future, or not in W3C Datetime format. Only the first two are common, and both make the value useless to Google.
Common causes
- A CMS that adds every published URL, including ones an SEO plugin sets to noindex.
- A migration that changed URLs while the sitemap generator kept the old ones, which now redirect.
- Parameter or filtered URLs in the sitemap that robots.txt blocks.
- Tag, author or paginated archives listed in the sitemap but canonicalized to another page.
- A generator that writes the build time into every lastmod.
- Deleted pages still listed and now answering 404.
Which URLs should be in a sitemap?
The ones you want in Google’s results, at the address you want indexed. Google’s sitemap guide says to include the URLs you want to see in search results and to use fully-qualified, absolute URLs, and it treats sitemap inclusion as a canonical signal. A URL that redirects, returns an error, says noindex or names another page as canonical sends the opposite message, which is what this audit flags.
Why flag a URL that robots.txt blocks?
Because Google cannot crawl it, so the sitemap is listing a page Google is told not to fetch. It also hides any noindex on that page: Google’s documentation says a noindex is only seen if robots.txt allows the crawl. The audit evaluates each URL against its host’s robots.txt for Googlebot specifically, so a rule that only applies to other crawlers is not flagged.
Does Google use lastmod?
Google says it uses lastmod when the value is consistently and verifiably accurate, for example when it matches when the page last changed, and that it ignores priority and changefreq. A sitemap that stamps the same date on every URL, or dates pages in the future, is not accurate, so the audit reports both.
How are 200 URLs chosen from a larger sitemap?
Evenly across the whole file, not the first 200. Large sitemaps are usually generated section by section, so the first 200 entries tend to be one section. For a sitemap index, up to five child sitemaps are read, spread across the index, and the sample is drawn from all of them. The lastmod checks run on every entry read, not just the sample.
Will this audit hammer my server?
No. At most three requests are in flight, request starts are spaced 300 milliseconds apart, and the whole audit stops at 90 seconds. If the site answers 429, or a bot-management layer refuses us, the audit stops at once and marks the rest not checked. Every request carries our published stillindexedbot user agent, and URLs robots.txt disallows for crawlers in the * group are not fetched.
Why is this check separate from the other tools’ hourly limit?
The other free tools share 60 fetches an hour per IP, and one audit can make more than 200. Rather than let one audit use up every other tool, audits have their own allowance of two an hour per IP, and at most two run on our side at once.
Does a clean audit mean every URL is indexed?
No. It means each sampled URL is fetchable, answers 200, does not say noindex, is not blocked for Googlebot, and names itself as canonical. Whether Google indexes it is Google’s decision, reported in Search Console’s Page indexing report. A clean audit removes the reasons that are under your control.