cPanel hosting

Host default robots.txt: the stray Disallow

A leftover or copy-pasted robots.txt in public_html disallows crawlers on a new site. Where the file lives, what the safe version is, and how to test it.

On cPanel hosting the live robots.txt is a physical file in public_html, and a leftover, copy-pasted, or default Disallow: / rule in it blocks every crawler from a new site. Open File Manager, read public_html/robots.txt, replace it with an allowing version, and test the live file again.

Why cPanel hosting does this

cPanel does not generate a robots.txt for you: the file exists only if someone or some script put it in the document root, which is public_html for the primary domain and a subdirectory per addon domain. That is exactly why a stray file survives: nobody remembers creating it, because nobody did. The routes in are mundane. A migration copied the old site's file, which was disallowing a staging host. A snippet pasted from a tutorial or from cPanel's own support guidance, which shows "User-agent: Googlebot / Disallow: /" examples for limiting crawlers, went in with "Disallow: /" left in it. Or a maintenance plugin wrote it during a rebuild and the site shipped before it was removed. Because robots.txt changes take effect without a deploy, the file can also change after launch while every page still returns 200.

Check it right now

Before changing anything, confirm what a crawler actually sees. The check is free, takes one URL and needs no account.

Run the check

How to fix it

  1. Run the robots.txt tester on the affected URL and copy the exact rule that matches it.
  2. Log in to cPanel and open File Manager.
  3. Navigate to public_html, or the addon domain's document root, and open robots.txt. If File Manager shows nothing, enable Show Hidden Files; the file may still be served by a rule higher up the server.
  4. Remove the matching "Disallow" or replace the file with "User-agent: *" and "Allow: /" plus any intentional rules.
  5. Save and run the tester again on the live URL until the block is gone.
  6. Do not add noindex to robots.txt; Google does not support it there, and a crawler blocked by robots.txt cannot see a page-level noindex either.

Why it happens again

The file is a plain text file at the edge of the server, editable by every person and plugin with file access, and it applies instantly with no build or review. Migrations, "block bots" tutorials, and maintenance modes all write to it, and nothing records who changed it last.

stillindexed re-checks the URLs you give it every 30 minutes on Starter and alerts when a directive changes, at most 30 minutes after it does. It is a monitor rather than a crawler: it watches a list you choose and tells you when one of seven things changes. Start with a 30-day free trial, 3 sites, no card required, and a 30 day refund after that.

Start the 30-day free trial, 3 sites, no card required

Catching it next time

Fixing it once is the easy half. The setting that caused this can be changed again by anyone with access, and the page will keep returning 200 while it happens.

robots.txt blocking pages that should be crawled, on other platforms

How Search Console reports it

Sources

Every claim about cPanel hosting above is from their own documentation, read on 2026-10-25. Platforms change their settings; if one of these is out of date, their page wins and we would like to know.