How can I monitor robots.txt for changes that block Google?
Monitor robots.txt by saving the paths that matter, fetching the site file on a schedule, and evaluating each path for Googlebot rather than comparing text alone. stillindexed fetches one robots.txt per site daily, marks a newly disallowed monitored path critical, confirms it on the next successful daily reading, then queues the configured alert.
Does the monitor fetch robots.txt once for every URL?
No. One HTTPS request fetches /robots.txt for the site. The parsed rules are then evaluated against every monitored path for both Googlebot and the wildcard agent. The product stores a separate permission result for each saved path, without fetching the same site file once per URL.
How often is robots.txt checked?
Robots.txt is part of the daily site-level run on every entitled, unpaused site. The 30-minute Starter and 15-minute Agency entitlements schedule monitored page checks, not robots.txt. A newly detected block confirms and enters the alert pipeline only if it still holds on a later successful daily reading.
This means a new robots block is not an immediate alert. It needs two successful site-level readings, and those readings are daily.
What robots.txt change is treated as critical?
An allowed-to-disallowed result for Googlebot creates a critical robots_blocked change. If Googlebot stays unchanged but the wildcard result becomes disallowed, that is critical too. Googlebot takes precedence when the two disagree. A monitored path already blocked on its first successful reading is judged by the same rule.
Will every text edit to robots.txt send an alert?
No. If the file text changes but every monitored path keeps the same permission, stillindexed records robots_changed as information. Information changes are stored as suppressed history and never alert. The alerting rule is based on the effective permission for a monitored path, rather than any textual difference in the file.
What happens when robots.txt cannot be read?
A failed fetch or parse does not become an allowed reading, and it cannot resolve an open robots incident. It raises robots_unreadable as a warning instead, because a signal you pay us to watch going dark is worth saying out loud. It holds until the file can be read again. A 404 is handled differently: no robots.txt means the monitored paths are allowed.
Can I check the current robots.txt without monitoring it?
Yes. The robots.txt tester fetches the live file and evaluates the paths you enter for Googlebot and the wildcard agent. It answers what the file means now. It does not schedule another check, so it is a current-state test rather than ongoing monitoring.
Which plan includes robots.txt monitoring?
Starter is $39 a month for up to 15 sites. Its monitored page cadence is 30 minutes, while robots.txt remains on the daily site-level schedule. Every entitled, unpaused site is eligible for that daily run.