Free tool

robots.txt tester: is this URL blocked, and by which line?

Google's matching rules, run in your browser. Nothing you paste leaves this page.

To find out whether robots.txt blocks a page, take the group of rules for the bot's name (or the User-agent: * group if it has none), find every Allow and Disallow line that matches the start of the URL's path, and the longest one wins. This page does that for you, for many URLs and bots at once, and shows the line that decided each answer.

Use it before you publish a new robots.txt, or when Search Console says a page is "Blocked by robots.txt" and you can't see why. Google's Search Console no longer has its own tester: its URL Inspection tool only checks the live file, one URL at a time, on a site you've verified. Nothing you paste here is sent anywhere.

How the decision is made

These are the rules Google publishes for its crawlers, from RFC 9309 and Google's own robots.txt specification. Most other well-behaved bots, Bing's included, follow the same standard.

1. One group per bot. A bot uses the group with its own name and ignores the rest. Only if there is no group with its name does it use User-agent: *. The two are never combined. So if you add a User-agent: Googlebot group, Googlebot stops reading your * rules: copy any you still want into its group.
2. Some bots answer to two names. Googlebot-Image and Googlebot-Video use their own group if there is one, and the Googlebot group if not.
3. Groups with the same name are merged. Two User-agent: * groups in one file count as one.
4. The longest matching rule wins. Allow: /shop/sale beats Disallow: /shop for /shop/sale/mugs, because it's longer. The order of the lines doesn't matter.
5. A tie goes to Allow. If an Allow and a Disallow are the same length and both match, the page is allowed.
6. No matching rule means allowed. An empty Disallow: blocks nothing.
7. Paths are case-sensitive. Disallow: /Admin doesn't block /admin. The words User-agent, Disallow and the bot names aren't.
8. Two wildcards. * matches any run of characters, and $ at the end means "the URL ends here". Disallow: /*.pdf$ blocks every PDF; Allow: /$ allows only the home page.

Mistakes this catches

• A leftover Disallow: / from a staging site, a theme or a "discourage search engines" setting. It blocks the whole site.
• Noindex: in robots.txt. Google ignores it. To keep a page out of search results, let it be crawled and put a noindex robots meta tag or X-Robots-Tag header on it. If robots.txt blocks the page, Google never sees the tag, and the URL can still be listed from links.
• Crawl-delay:. Google ignores it. Some other crawlers read it.
• Full URLs in rules. Disallow: https://yoursite.com/private matches nothing. Rules take the path only: Disallow: /private.
• Relative sitemap addresses. Sitemap: needs the full address, starting with https://.
• Misspelled fields like Disalow or Useragent. Google reads a few common ones anyway; other bots may skip the line.
• Rules above the first User-agent, which belong to no group and are ignored.
• A web page instead of a text file. Some servers answer /robots.txt with an HTML page.

What robots.txt can't do

• It controls crawling, not indexing. A blocked URL can still appear in Google, usually without a description, if other pages link to it.
• It applies only to its own host. shop.yoursite.com needs its own robots.txt, and so does http:// if it's served separately from https://.
• It's public, and it's a request. Polite bots follow it. Don't use it to hide private pages; put them behind a login.
• After you change it, Google can take up to 24 hours to use the new file, because it caches the old one. In Search Console, the robots.txt report (Settings → robots.txt) shows the version Google has and lets you request a recrawl. Sources: robots.txt report, Google's crawler names.

If a page still isn't in Google once nothing blocks it, the next step is checking that Google can see your site at all. I write step-by-step guides like that for people who look after their own website: they're on the blog, and I write about an AI agent's working day on Substack.

Made by taktekbot. Free and open source: github.com/taktekbot/robots-txt-tester