Noindex vs disallow: which keeps a URL out of Google

Disallow blocks crawling, noindex blocks indexing. A disallowed URL cannot pass its noindex to Google, so pick the mechanism per URL type with the tables here.

Disallow controls crawling. noindex controls indexing. A disallowed URL cannot have its noindex read, so combining the two on one URL can keep a page in Google while you believe you removed it. Pick by goal: noindex to remove a URL from results, Disallow to manage crawl access, and never both on the same URL at the same time.

The short comparison

robots.txt Disallow noindex meta tag X-Robots-Tag header
What it controls Crawl access to a path Indexing of an HTML page Indexing of any response
Where it lives /robots.txt at the host root <meta name="robots" content="noindex"> in the head HTTP response header
Google must fetch the URL No, it reads robots.txt instead Yes Yes
URL leaves results Not reliably Yes, after a recrawl Yes, after a recrawl
Search Console report Blocked by robots.txt, or Indexed, though blocked Excluded by noindex tag Excluded by noindex tag

The row that matters is the third one. noindex is an instruction printed on the page or in the response. If Google is not allowed to fetch the response, the instruction is invisible.

What each mechanism actually does

robots.txt is read before any fetch. A Disallow line removes crawl access, and Google is direct that this is not a removal tool: “it is not a mechanism for keeping a web page out of Google. To keep a web page out of Google, block indexing with noindex or password-protect the page” (robots.txt introduction).

A blocked page can still surface. “its URL can still appear in search results, but the search result won’t have a description.” Links from elsewhere create a second path: “we might still find and index a disallowed URL if it is linked from other places on the web.” When that happens, “the URL address and, potentially, other publicly available information such as anchor text in links to the page can still appear in Google Search results.”

noindex works because it arrives with the page. “When Googlebot crawls that page and extracts the tag or header, Google will drop that page entirely from Google Search results, regardless of whether other sites link to it” (block indexing). That sentence also explains the dependency: no crawl, no tag, no removal.

The mechanism is not only a meta tag. “A response header can be used for non-HTML resources, such as PDFs, video files, and image files.” A PDF cannot carry a meta tag in its head, so the header is the only noindex option there. Test what a URL returns with the X-Robots-Tag checker and read the full header guide before you touch server config.

Google’s 2019 note adds the reason Disallow appears to work in the short term: “Search engines can only index pages that they know about, so blocking the page from being crawled usually means its content won’t be indexed” (unsupported rules in robots.txt). The word usually does the work. Known URLs, especially linked ones, break the assumption.

How each shows up in Search Console

There are three distinct outcomes, and they map to three different mistakes.

Excluded by noindex tag. Google fetched the URL, saw the tag or header, and dropped it. This is the state you want while a page is being removed. Check it in the noindex report.

Blocked by robots.txt. Google did not fetch the URL. This report tells you a rule exists, not that the URL is gone: “Note that this does not guarantee that the page won’t be indexed through some other means” (Page indexing report). Inspect it in the blocked report.

Indexed, though blocked by robots.txt. This is the trap made visible. “The page was indexed despite being blocked by your website’s robots.txt file.” Google explains the path: “Google won’t request and crawl the page, but we can still index it, using the information from the page that links to your blocked page.” Snippets suffer: “any snippet shown in Google Search results for the page will probably be very limited.” The report lives at Indexed, though blocked by robots.txt.

URL Inspection has a related quirk. A blocked URL can report “Indexing allowed? Yes” even while the block is active. Google’s help page warns: “When this happens, your page might appear in Search results even if indexing is blocked by your site” (URL Inspection). Read that field alongside the robots.txt row, not instead of it.

Which mechanism fits your scenario

Scenario Right mechanism Wrong mechanism and its failure
Staging site that must stay out of the index noindex meta or header, plus password protection if it holds client data Disallow alone: linked URLs can still be indexed and appear without a description. See staging site indexed
Thin faceted URLs, such as filter parameters you never want indexed noindex on the response, or a canonical if the content is a duplicate Disallow: the URLs stay eligible for indexing from internal links, and you lose the ability to see the noindex state
PDFs, video files, image files X-Robots-Tag noindex header Meta robots: the file has no HTML head. Disallow: the file remains indexable if linked
Private pages with confidential content Password protection or authentication robots.txt: the RFC is explicit that “These rules are not a form of access authorization” (RFC 9309)
Internal search results pages noindex Disallow: internal links keep feeding Google the URLs, which can then be indexed from those links
A page already indexed that you want gone Keep it crawlable and serve noindex until it drops, and leave it crawlable Disallow first: the noindex is never read and the page can stay indexed

The trap: adding Disallow on top of noindex

The instinct is reasonable. A page is indexed, so you add noindex, then add Disallow to stop Google from crawling it while the removal happens. The second step cancels the first. “For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawler.” If it is blocked, “the crawler will never see the noindex rule, and the page can still appear in search results, for example if other pages link to it.”

The correct order is one direction only: crawlable first, noindex second, removal third. Adding Disallow afterwards reopens the same gap, because Google can no longer see the noindex. Google states the fix plainly: “To avoid being indexed, remove the robots.txt block and use ‘noindex’.” Its own FAQ repeats the warning: “Using a robots.txt rule is not recommended for blocking a page, and will actually prevent noindex from being seen by Google.”

There is a second cost. While a URL is disallowed, you cannot confirm its noindex state, because Google cannot fetch it. You are monitoring a blind spot. Keep the URL fetchable until the Excluded by noindex tag count includes it.

The removal order when a URL is already indexed and disallowed

  1. Establish the state. Check URL Inspection and the Page indexing report. If the status is “Indexed, though blocked by robots.txt”, both problems exist. Run the indexability checker against the live URL to see what a plain HTTP client receives.
  2. Remove the Disallow line for that path only. Keep the rest of robots.txt intact so you do not open crawl paths you closed on purpose. Test the file after the edit with the robots.txt tester.
  3. Serve noindex on the URL. Use the meta tag for HTML and the X-Robots-Tag header for anything else, then verify the header with the X-Robots-Tag checker.
  4. Wait for a recrawl. Google has to revisit the URL: “it may take months for Googlebot to revisit a page.” Watch the noindex report rather than guessing.
  5. Once the URL is out, leave it crawlable with noindex. Re-adding Disallow hides the noindex from Google again, and Google says a disallowed URL that is linked from elsewhere can still be indexed, so the URL can come back as a result without a description. If crawl load forces a Disallow, watch the Indexed, though blocked by robots.txt count afterwards.

The Removals tool buys time, not a fix

For an urgent takedown, the Removals tool hides a result faster than noindex can. Treat it as temporary: “A successful request lasts only about six months.” It also does not change crawl behavior: “Blocking a URL does not prevent Google from crawling your page, only from showing it in Search results” (Removals tool).

Two operational details catch people. Permanent removal options are a 404 or 410, a password, or noindex, and Google adds that noindex “is less secure than the other methods” and says “Do not use robots.txt as a blocking mechanism.” A removal request also expires early if the URL cannot be reached: “If your URL is unreachable by Google (404, 502/3) when you use this tool, it will assume that the page is gone, and your block request will expire.”

For genuinely private content, skip the search removal logic and use access control: “If you have confidential or private content on your site, you need to password protect it to ensure only authorized users can access it” (control what you share).

noindex in robots.txt is not a rule

Some robots.txt files contain a Noindex line. It does nothing: “Specifying the noindex rule in the robots.txt file is not supported by Google” (block indexing). Google also warns against the general pattern: “Don’t use a robots.txt file as a means to hide your web pages (including PDFs and other text-based formats supported by Google) from Google Search results.” Compliance is voluntary in any case: “it’s up to the crawler to obey them.”

If you find a Noindex line in a client’s robots.txt, remove it and replace the intent with a real noindex on the responses that need it, or with authentication if the content is private.

Checking and monitoring both states

The two mechanisms fail in opposite directions. A noindex that nobody notices removes a page from results. A Disallow that nobody notices hides the fact that a noindex was never read. Both are worth a recurring check rather than a one-off audit. A single URL can be tested in seconds with the indexability checker, and a whole robots.txt file with the robots.txt tester.

For client sites, stillindexed re-checks monitored URLs every 30 minutes on Starter or every 15 minutes on Agency, reads robots.txt hourly, and confirms a new block on a monitored URL with a second read about five minutes later before alerting. It fetches over plain HTTP with no JavaScript rendering, so header-level noindex and robots rules are visible without Search Console access. The trial is 30 days, 3 sites, no card required, with plans on the pricing page.

Sources

Questions

Does noindex work if the URL is disallowed in robots.txt?
No. If a URL is disallowed, Google does not fetch it, so it never reads the noindex tag or header. The page can still be indexed from links pointing at it, and Search Console reports it as indexed though blocked by robots.txt. Remove the Disallow rule first if you need the noindex to take effect.
Which is better for keeping a page out of Google, noindex or disallow?
Use noindex when the goal is removal from search results, including staging sites, thin pages, and internal search pages. Disallow is for crawl management, such as keeping a crawler out of a directory you do not need fetched. Google states that robots.txt is not a mechanism for keeping a page out of Google.
Can a disallowed URL still appear in Google?
Yes. Google can index a disallowed URL when other pages link to it, using the linking page for information. The result usually has no description or a very limited snippet. Google's documentation says a disallowed URL can still appear with its address and anchor text from inbound links.
How long does noindex take to remove a page from Google?
It depends on recrawl frequency. Google has to fetch the page, see the noindex, and then drop it. Google's documentation notes it may take months for Googlebot to revisit a page. For faster, temporary hiding, a Removals tool request lasts about six months.