robots.txt Tester
Paste or fetch a robots.txt, validate its syntax, and test whether a given URL is allowed for a crawler — per RFC 9309.
The file at /robots.txt that tells crawlers which paths they may fetch.
/robots.txt is the file a site publishes to tell crawlers which paths they may or may not fetch. It's organized into groups: one or more User-agent lines followed by Allow / Disallow rules. A crawler picks the group whose user-agent token is the longest match for its name (falling back to *), then applies the most specific matching rule — by RFC 9309, the longest matching pattern wins, and Allow beats Disallow on a tie.
Patterns support * (any sequence) and $ (end of URL). This tester parses all of that in your browser, flags malformed directives, summarizes which agents and sitemaps are covered, and tells you exactly which rule decides a given URL. Use "Fetch" to pull a live site's robots.txt (fetched server-side, only the /robots.txt path).