The stillindexedbot

User agent

All requests are sent with this single fixed string:

Mozilla/5.0 (compatible; stillindexedbot/1.0; +https://stillindexed.com/bot)

Whitelist it in your logs, WAF and analytics instead of our IPs.

Egress IPs

Every request comes from one of these addresses:

• 89.167.111.222

If you see 403s, 429s or a WAF challenge, allowlisting the user agent is the reliable fix.

When your edge turns us away we say so. A 403 that carries a bot-management fingerprint, or that a browser-shaped request gets past, is reported to the owner ascrawler blocked at warning level, not as their page being broken. Working out which of the two it is costs one extra request, sent only after a 403 or 429.

Politeness rules

• 2 seconds between requests when we check several URLs on the same site.
• One GET per check. No JavaScript is executed, so nothing runs your analytics tags.
• Maximum body 2 MiB, request timeout 15 seconds, at most 10 redirect hops.
• We never send cookies and never log in. We only see what an anonymous visitor sees.
• How often we fetch is your cadence, not ours: every 30 minutes on Starter, every 15 on Agency. That is 48 or 96 requests a day for one monitored URL.

Honest limits: two different accounts monitoring the same host are not yet coordinated, so they can overlap. We do not currently send conditional requests, so each check is a full GET rather than a cheap 304.

How we treat robots.txt

We read your robots.txt on every site-level check and evaluate it per monitored path, for the wildcard group and for Googlebot. That evaluation is a monitored signal: a path flipping from allowed to disallowed is one of the things we alert you about.

We do not use robots.txt to decide whether to fetch a URL you are monitoring. Two reasons, both worth stating plainly. You added that URL to your own account, which is the authorisation. And a page becoming disallowed is exactly the disaster this product exists to catch, so a crawler that stopped looking the moment a rule appeared would go silent at the precise moment it mattered.

If you would rather we did not fetch a URL at all, remove it from the account monitoring it, or block the user agent above at your edge. We treat a block as your answer and do not route around it.

Allowlisting in three commands

# robots.txt: allow us explicitly
User-agent: stillindexedbot
Allow: /

# WAF / firewall: allow our published egress IPs (see above)
# Cloudflare security level -> skip or challenge

Questions or a misbehaving check? The bot line on this page is the canonical contact; it links here so you can see everything about the request in one place.

Other bots in your logs

Most of the crawler questions we get are not about us. If you are working out what GPTBot, ClaudeBot, PerplexityBot or Google-Extended are for, and what blocking each one actually costs you, we wrote that up with the behaviour taken from each operator's own documentation: how AI crawlers see your site.

To see whether a crawler and a browser get the same page from you, thecrawler versus browser checker fetches one URL as Googlebot, Bingbot, GPTBot and a browser and reports where the responses differ.