stillindexedStart monitoringOpen the app

Netlify

Netlify robots.txt blocking pages: fix it

Find the robots.txt in Netlify's deployed output, remove the bad rule at its build source, and publish the correction.

Netlify serves only the robots.txt copied or generated into the publish directory, so editing another repository file changes nothing. Inspect the latest production output in "Deploy File Explorer," trace the unwanted Disallow to its source or build step, remove it there, deploy again, and test the live file.

Why Netlify does this

Netlify deploys only files in the configured publish directory. A robots.txt copied or generated into that directory becomes the live file, while a robots.txt elsewhere in the repository has no effect. The deployed file can be inspected under Deploys in the "Deploy File Explorer". Its Disallow rules control crawling, not guaranteed indexing, so Google may still list a blocked URL without its page content.

Check it right now

Before changing anything, confirm what a crawler actually sees. The check is free, takes one URL and needs no account.

Run the check

How to fix it

  1. Run the robots.txt tester on the affected URL and copy the matching rule.
  2. In Netlify, open the project, choose "Deploys", select the latest published production deploy, and find robots.txt in the "Deploy File Explorer".
  3. If the file is absent or differs from the repository copy, check the "Publish directory" under Project configuration > Build & deploy > Continuous deployment > Build settings.
  4. Edit the source file or build step that writes robots.txt into that publish directory. Remove only the matching "Disallow".
  5. Trigger a production deploy, inspect the new deployed file, then run the tester again against the public URL.
  6. Do not rely on noindex while the URL is blocked. Google must crawl the page or response header to see that directive.

Why it happens again

A generated robots.txt is recreated on every build. Editing a generated output locally, or editing a source file outside the publish directory, appears fixed in the repository but leaves the deployed response unchanged.

stillindexed re-checks the URLs you give it every 30 minutes on Starter and alerts when a directive changes, at most 30 minutes after it does. It is a monitor rather than a crawler: it watches a list you choose and tells you when one of seven things changes. Card first, no trial, and a 30 day refund.

See what monitoring covers

Catching it next time

Fixing it once is the easy half. The setting that caused this can be changed again by anyone with access, and the page will keep returning 200 while it happens.

Other ways Netlify loses pages

robots.txt blocking pages that should be crawled, on other platforms

Sources

Every claim about Netlify above is from their own documentation, read on 2026-08-30. Platforms change their settings; if one of these is out of date, their page wins and we would like to know.