Webflow
Webflow robots.txt blocking pages: fix it
Remove an accidental Disallow from Webflow Site settings, publish it, and verify the rule on the live domain.
Webflow publishes custom robots rules from "Site settings > SEO > Indexing," so a broad Disallow there can block intended pages until the site is republished. Remove the matching rule, save and publish the site, then test robots.txt on the custom domain to confirm the block has gone.
Why Webflow does this
Webflow puts custom robots.txt rules at Site settings > SEO > Indexing. A broad "Disallow: /" blocks the whole site, while a path rule can block one page or folder. The setting is not live until the site is published. Disallow controls crawler access; it does not guarantee removal from Google, and a blocked URL can still appear without a description.
Check it right now
Before changing anything, confirm what a crawler actually sees. The check is free, takes one URL and needs no account.
How to fix it
- Run the robots.txt tester on the affected URL and copy the rule that matches it.
- Open Site settings > SEO > Indexing.
- Remove the matching "Disallow" from the robots.txt rules. Keep unrelated rules intact.
- Click "Save and publish your site".
- Run the tester again on the custom domain and confirm the published response no longer blocks the page.
- If the goal is to remove a page from Google, use a page-level noindex after crawling is allowed. Google cannot see noindex while the URL remains blocked.
Why it happens again
The text box is site-wide and changes take effect only after publishing. A saved correction can look complete in Site settings while the old robots.txt remains live.
stillindexed re-checks the URLs you give it every 30 minutes on Starter and alerts when a directive changes, at most 30 minutes after it does. It is a monitor rather than a crawler: it watches a list you choose and tells you when one of seven things changes. Card first, no trial, and a 30 day refund.
Catching it next time
Fixing it once is the easy half. The setting that caused this can be changed again by anyone with access, and the page will keep returning 200 while it happens.
Other ways Webflow loses pages
- my Webflow staging site is indexed by Google
- Pages, CMS items or the staging site are missing from Google after a Webflow publish.
- A Webflow page tells Google the wrong URL is its preferred version after a domain change, a campaign override or a second tag from a tag manager.
- my Webflow migration lost pages from search
robots.txt blocking pages that should be crawled, on other platforms
Sources
Every claim about Webflow above is from their own documentation, read on 2026-08-30. Platforms change their settings; if one of these is out of date, their page wins and we would like to know.