Fixing indexing problems, platform by platform

The same problem has a different cause on every platform, and the cause is the part you need. Each page names the exact setting, links to a free check, and cites the vendor's own documentation.

If you do not yet know which of these it is, start withthe six checks that rule each one out. If you are starting from a status in Search Console's Page indexing report, there isa page for each status, named exactly as the report names it.

A staging or preview site indexed by Google

  • VercelVercel protects normal previews, but a custom domain on a non-production branch can lose noindex. Check the header and restore it.
  • NetlifyNetlify can index the newest branch deploy. Add a branch-only X-Robots-Tag without blocking the production site.
  • WebflowStop a webflow.io staging site appearing in Google by turning off Webflow staging indexing and removing the existing result.
  • WordPressStop a public WordPress staging site appearing in Google with the Reading setting that adds a robots noindex tag.
  • VercelVercel noindexes Preview Deployments and outdated production deploys by header. Where the header comes from, the custom domain exception, and the fix.
  • Cloudflare PagesPreview deployments carry X-Robots-Tag: noindex by default, but branch aliases on custom domains may not. The pages.dev versus custom domain trap.
  • RenderRender does not add X-Robots-Tag: noindex to previews automatically. How to add response headers yourself, per service type, and verify.
  • RailwayRailway sends no noindex header on staging or PR environment domains. What the platform actually documents, and how to keep those URLs out of Google.

A noindex tag hiding the site from search

  • WordPressThe Discourage search engines box writes a noindex meta tag into every page. The exact setting, how to undo it, and why it comes back.
  • ShopifyShopify hides pages with Liquid conditions in theme.liquid and the seo.hidden metafield. The exact steps, and how to check a page right now.
  • SquarespaceA site-wide crawlers switch and a per-page Hide page from search results toggle. The exact paths, the plan limits, and why it comes back.
  • WebflowWebflow's per-page Sitemap indexing toggle writes a noindex meta tag. The exact menu paths, the staging subdomain switch, and the trap.
  • WixThe Let search engines index this page toggle sits in each page's SEO basics settings. Exact steps, and the site-wide switch that overrides them.
  • WebflowThe Disable search engine indexing switch sits in Site settings > SEO > Indexing and noindexes the published site. The checkbox people miss, and the fix.
  • WixThe "Let search engines index your site" switch in the SEO dashboard overrides every page toggle. The Hide Site step people miss, and how to undo it.
  • SquarespaceSite Availability set to Private or Password Protected hides the whole site from search engines. The panel, the per-page SEO toggle, and how to go public.
  • GhostPrivate site mode switches off every SEO feature sitewide, while members-only posts stay indexable as URLs. Where each switch lives, what each one costs.
  • GoDaddyThe Websites + Marketing builder separates editing from publishing, and unpublishing swaps the site for Coming Soon. The Settings panel and going live.
  • Staging environmentsOne environment variable decides where the noindex header or meta tag goes, and a wrong value noindexes production. How the leak happens, how to trace it.
  • NetlifyNetlify noindexes Deploy Previews and old branch deploys by header, and the _headers file is global. Which deploys get noindex and how to scope your own.
  • FramerFramer writes a noindex meta tag from a per-page setting, and publishing is a separate action from editing. Both settings, both failure modes.

robots.txt blocking pages that should be crawled

  • ShopifyFind the live robots.txt.liquid rule, remove an accidental Disallow, and restore Shopify defaults when the whole custom file is wrong.
  • WordPressWork out whether WordPress, a plugin, or a physical robots.txt owns the live rule, then remove the block from the copy that is served.
  • WebflowRemove an accidental Disallow from Webflow Site settings, publish it, and verify the rule on the live domain.
  • SquarespaceUndo the site-wide crawler block, or identify a built-in Squarespace exclusion that the platform does not let you edit.
  • NetlifyFind the robots.txt in Netlify's deployed output, remove the bad rule at its build source, and publish the correction.
  • ShopifyPrivate mode hides the store from visitors and search engines and the password page ships Disallow rules. The launch checkbox people miss, and the fix.
  • cPanel hostingA leftover or copy-pasted robots.txt in public_html disallows crawlers on a new site. Where the file lives, what the safe version is, and how to test it.
  • CargoCargo saves every change as a draft until you publish, and sites need a paid subscription to go public. The two gates that keep a portfolio out of Google.
  • ShopifyShopify generates robots.txt from a Liquid template you can add. What the default disallows, how to extend it without breaking it, and how to undo.

A canonical tag pointing at the wrong URL

  • ShopifyShopify emits canonicals through the theme's canonical_url object. Where the tag comes from, what breaks it, and how to check any page.
  • WordPressYoast and Rank Math both write canonicals and either can override the other. The exact fields, the filters, and how to check a page.
  • WebflowWebflow fills canonicals automatically, and three documented mistakes break them: a stale global tag URL, a page override, or a duplicate tag.
  • WixWix creates a default canonical for every page, and the Advanced SEO tab can override it. Where the override lives, and how to check a page.

A migration or redesign losing pages from search

  • ShopifyMap old store URLs to their new Shopify paths, add the missing redirects, and test the URLs that carried search traffic.
  • WordPressSeparate WordPress old-slug redirects from a full migration map, then restore the missing server redirects without changing every URL again.
  • WebflowUse Webflow automatic slug redirects where they apply, then map folders, collections, and localized paths in the 301 redirect table.
  • SquarespaceTranslate old page and collection paths into Squarespace URL mappings, including blog and store patterns that change every item URL.

A broken or chained redirect

  • CloudflareCloudflare redirects at the edge, your origin redirects too, and the hops stack. The exact settings, the documented loop causes, and how to count hops.
  • NetlifyNetlify redirects apex to www or the reverse automatically, and your _redirects rules add hops on top. The exact mechanisms and the loop trap.
  • VercelVercel redirects www automatically even with no rule configured. The domain redirect, the project redirects, and how to count the hops.
  • WordPressWordPress issues its own 301 to the address configured in Settings > General. The redirect_canonical mechanism, the two fields, and the fix.
  • ShopifyShopify redirects every listed domain to the primary automatically, and manual URL redirects can add a second hop. The exact settings.

An expired or expiring TLS certificate

  • CloudflareCloudflare renews Universal SSL certificates automatically, and the documented failures are DCV and CAA. The windows, the causes, and the checks.
  • NetlifyNetlify provisions and renews Let's Encrypt certificates automatically. The retry schedule, the four documented failure causes, and the fixes.
  • VercelVercel generates certificates for every domain automatically, but uploaded certificates are your problem. The documented 5 day fallback.
  • ShopifyShopify issues TLS certificates automatically and blocks uploaded certificates. The 48 hour window, the CAA list, and the documented fixes.
  • WordPress on cPanelcPanel's AutoSSL installs and renews certificates nightly for WordPress sites. The mechanism, the CAA preflight, and how to force a check.

An expired or expiring domain registration

  • Cloudflare RegistrarCloudflare Registrar auto-renews by default, first attempt 30 days out. The documented failure causes, the notices, and the grace period.
  • NamecheapNamecheap parks expired domains and keeps them reactivate for 30 days. The notices, the grace period, and how to reactivate from the account.
  • GoDaddyGoDaddy parks expired domains at five days and auctions them later. The documented timeline, the auto-renew attempts, and how to recover.
  • Squarespace DomainsSquarespace domains auto-renew and expire on failed charges. The grace period is 30 days on Squarespace LLC registrars, 40 on Tucows.

An AI search crawler blocked from a site

  • CloudflareCloudflare blocks AI crawlers by default on new domains. How to allow OAI-SearchBot, PerplexityBot and Claude-SearchBot while still blocking training.
  • OpenAIGPTBot and OAI-SearchBot are separate robots.txt tokens. Blocking GPTBot keeps you in ChatGPT search; blocking OAI-SearchBot removes you.
  • PerplexityPerplexityBot surfaces pages in Perplexity answers and does not train models. A robots.txt or WAF block removes you; Perplexity-User ignores robots.txt.
  • GoogleGoogle-Extended is a control token, not a crawler. Blocking it stops Gemini training and grounding, not Google Search or AI Overviews.