robots.txt Disallow vs noindex: Why Blocked Pages Still Show

PublishedToolrobots.txt Generator

Contents
  1. Why does a blocked page still show up in search?
  2. Crawling and indexing live on different layers
  3. Which one should you use?
  4. How to check and write it yourself, without a tool
  5. Common stumbling blocks
  6. The bottom line
  7. Frequently asked questions

You added Disallow to robots.txt to keep a page out of search results, but days later that URL is still showing up. Nothing is misconfigured — Disallow simply doesn't do that job.

robots.txt is the file that controls whether robots read a page (crawling). Whether a page gets listed in search results (indexing) is decided by noindex. They're two different layers, so one can't stand in for the other.

Summary ① Disallow is a crawling block signal. Google's documentation states that a page blocked by robots.txt can still be indexed if other sites link to it. ② When that happens, the URL appears in search results with no description. ③ To get a page out of search, you need noindex — but if robots.txt blocks the page, robots can't see that noindex. Don't apply both.

To quote Google Search Central's documentation directly: a page disallowed in robots.txt can still be indexed if other sites link to it. And those indexed results appear without a description.

The reason is simple. The robot couldn't read the page body, but it can still learn that "a URL like this exists" and see the anchor text of outside links pointing to it — robots.txt has nothing to do with that.

So the block is working exactly as intended, and the URL still stays in search. In the Page indexing report in Google Search Console, this shows up as the Indexed, though blocked by robots.txt warning.

In other words, "I blocked it, so why is it still there?" isn't a malfunction — it's the result of picking the wrong tool for the job.

Crawling and indexing live on different layers

Compare where each setting lives and what it blocks, and the difference becomes clear.

Disallow in robots.txtnoindex
What it blocksCrawling (robots reading the page)Indexing (being listed in search results)
Where you put itThe /robots.txt file at the site rootA meta tag on the page or an HTTP response header
What robots need to see the ruleJust read the robots.txt fileMust be able to crawl that page
A URL that's already indexedMay stay as isDrops out after it's crawled again

The third row of that table is the heart of this post. noindex is a signal written inside the page, so robots can only read it by opening the page.

That's why adding Disallow and noindex to the same URL at the same time actually keeps it indexed. Google's documentation states flatly that "for the noindex rule to work, the page must not be blocked by robots.txt."

Which one should you use?

The right tool depends on the outcome you want.

What you wantWhat to use
Reduce crawl trafficDisallow in robots.txt
Keep a page out of search resultsnoindex (with crawling left open)
Keep a page hidden from everyonePassword protection or deleting the page

Google recommends three ways to keep something out of search: password-protect the file, use a noindex meta tag or response header, or delete the page entirely. Blocking with robots.txt is not on that list.

One more thing worth noting: robots.txt is only a request to robots, and anyone can open the file itself. Listing sensitive paths under Disallow effectively publishes that list of paths.

How to check and write it yourself, without a tool

Start by checking the current state

Add /robots.txt after your domain in the browser address bar, and you'll see the file currently in effect. That file must live at the site root, named robots.txt in lowercase (RFC 9309); if you put it in a subfolder, it won't be recognized.

To check whether a page has noindex, open View Source in your browser and look for <meta name="robots" content="noindex">. Non-HTML files (like PDFs) set it through response headers, so run curl -I URL and check for X-Robots-Tag.

robots.txt syntax takes four lines

Google supports four fields: user-agent, allow, disallow, and sitemap. A group starts with User-agent, and the rules under it apply to that robot.

User-agent: *
Allow: /admin/help/
Disallow: /admin/
Disallow: /search

Sitemap: https://example.com/sitemap.xml
  • Leaving Disallow: empty means block nothing (the empty pattern in the RFC). That's how you express allow-all.
  • Block everything is Disallow: /. One slash makes it the exact opposite.
  • When rules overlap, the most specific one wins — that is, the rule with the longest path. If the lengths are equal, Allow takes priority.
  • Paths are case-sensitive. /Admin/ and /admin/ are different paths.
  • Crawlers must support * (any number of characters, including zero) and $ (end of path). You'd write it as Disallow: /*.pdf$.
  • Anything after # is a comment.

In the example above, /admin/help/ has a longer path than Disallow: /admin/, so it's more specific — which means you block the admin folder while leaving the help section open.

Where to put it and when it takes effect

The file applies per host, protocol, and port. blog.example.com and example.com each need their own robots.txt.

It doesn't take effect instantly, either. Google usually caches robots.txt for up to 24 hours, and RFC 9309 also advises crawlers not to use a cached file for more than 24 hours. Google also ignores anything beyond 500 KiB.

Build it with our tool. Once you're juggling multiple groups, mistakes creep in: blank lines between groups, the leading slash on paths, spelling of field names. With the tool, you enter User-agent and your Disallow/Allow paths (one per line) plus a Sitemap URL, and it assembles a spec-compliant file and shows it to you on the spot. There are Allow all and Block all presets, an add-User-agent-group button, and copy and file download buttons.

The tool also has a Crawl-delay field, but it's optional. Google doesn't include crawl-delay in its list of supported fields, so unless you need it for another search engine, you can leave it blank.

robots.txt Generator

It runs right in your browser, with no install and no sign-up. The paths you enter are never sent to a server.

Common stumbling blocks

If you want to remove a URL that's already indexed, order matters. You have to lift the Disallow first so robots can come in, read the noindex, and drop the page from the index. Adding noindex while the block is still in place does nothing.

SituationWhat happens
Writing a Noindex: line in robots.txtIt's a field Google doesn't support, so it's ignored
Applying Disallow + noindex togetherThe noindex is never read, so the page stays indexed
Setting Crawl-delayGoogle doesn't support it
Blocking your CSS and JS foldersYou also block files needed to render the page, which can change how it's evaluated
Listing sensitive paths under DisallowThe file is public, so those paths are exposed as is

Crawl-delay isn't a field defined in RFC 9309, and Google doesn't include it among supported fields. Some other search engines do interpret it, though, so if you need it, check that engine's webmaster documentation to confirm support.

When you need a result taken down fast, you can use the removal tool in Search Console — but that's a temporary measure. Google's documentation says requests last about 6 months, so in the meantime you need to set up a permanent fix, like noindex or deleting the page.

The bottom line

  • Disallow in robots.txt means don't crawl, not "exclude from search results."
  • A blocked URL can still get indexed if other sites link to it, and when that happens, only the URL shows up, with no description.
  • To keep a page out of search results, use the noindex meta tag or the X-Robots-Tag response header, and leave crawling for that page open.
  • Using both at once means noindex never gets read, so you get the opposite of what you wanted. If you've already blocked the path, unblock it first.
  • Anything that truly must stay hidden should go behind a password or be deleted. robots.txt is a public file.
  • Put the file at the top-level path as lowercase robots.txt. Rules follow longest path wins and are case-sensitive.
robots.txt Generator

It runs right in your browser, with no install and no sign-up. The paths you enter are never sent to a server.

Frequently asked questions

Can't I just put noindex in robots.txt?

Google states that it does not support specifying noindex in a robots.txt file. The only supported fields are user-agent, allow, disallow, and sitemap, and any other lines are ignored. noindex has to go in the page's meta tag or in an HTTP response header.

What order should I do things in to remove a URL that's already showing in search?

First, remove the Disallow for that path in robots.txt so crawlers can reach the page. Then add noindex to the page and wait for it to be crawled again. If you're in a hurry, you can use the removal tool in Search Console as a stopgap, but it only lasts about 6 months, so you need to finish the job with noindex or by deleting the page within that time.

I added Crawl-delay, but the crawl rate hasn't changed

That's because crawl-delay isn't one of the fields Google supports. It isn't defined in RFC 9309 either — it's a non-standard field, so whether it's honored depends on the search engine. Check the webmaster tool documentation for the search engine you're using to see whether it's supported, and if it isn't, your best bet is to look at server-side settings or the crawl rate settings each tool provides.

Do subdomains and http vs. https need their own files?

Yes. robots.txt applies per host, protocol, and port. The rules in https://example.com/robots.txt do not apply to https://blog.example.com, and each host needs its own file at its own top-level path. Addresses on a different port are separate too.

This article was adapted from our Korean original. Read the Korean original

Related articles

  1. Transparent PNG Signature: Draw on Canvas and SaveDraw your signature on a canvas and save it as a transparent PNG. How alpha channels work, and how image signatures differ from certified ones.
  2. Sitemap priority and changefreq are ignored: what lastmod meansGoogle's docs say it ignores sitemap priority and changefreq. See what the spec sets, when lastmod actually counts, and why a sitemap doesn't guarantee indexing.
  3. How to Split People Into Teams: Team Sizes and RemaindersSee how many people land in each team when the headcount doesn't divide evenly, and how the extra people are added to the first teams.
  4. Why You Shouldn't Use Real Customer Data as Test DataCopying production data spreads real customer info through dumps, logs, and reports. Learn to spot it and generate fake data instead.
  5. Username Already Taken? How to Make Nicknames That Don't OverlapWhy the username you want is always taken, and how to combine adjectives, nouns, and numbers to generate several non-overlapping nickname options at once.
  6. UUID v4 structure: 122 random bits and collision oddsSee why UUID v4 splits into 8-4-4-4-12, what the 4 and the 8/9/a/b mean under RFC 9562, and how likely a duplicate really is.

Related tools