🚦 robots.txt Tester

How a crawler decides

A robots.txt file is split into groups. Each group starts with one or more User-agent lines and lists Allow and Disallow rules. A crawler first picks its group: the one naming it exactly, otherwise the longest name it starts with (so Googlebot-Image follows a googlebot group), otherwise the * group. If several groups name the same crawler, their rules are combined.

Inside the group, the longest rule that matches the URL path wins. If an Allow and a Disallow rule are equally long, Allow wins. A * matches any characters and a $ marks the end of the URL. Paths are case-sensitive: Disallow: /fish blocks /fish.html but not /Fish.html.

What robots.txt does not do

Blocking a URL in robots.txt stops crawling, not indexing. Google can still list a blocked URL - without a description - if other pages link to it. To keep a page out of search results, let it be crawled and add a noindex meta tag instead. The noindex line inside robots.txt itself has not worked in Google since 2019, and this tester flags it.

Limits of this tester

It applies the published rules to the text you paste; it does not fetch your live file, so test what is actually on your server. Percent-encoded characters (like %2F) are compared as written, and crawl-delay is ignored, as Google ignores it. Some crawlers fetch a page when a person asks for it and say robots.txt may not apply to them - the AI Crawler Access Checker lists which.