robots.txt Tester
Fetch or paste a robots.txt and test whether Googlebot, Bingbot or AI crawlers may crawl a URL — with the matching rule.
Live check made by our server to the public address you enter (rate-limited).
How matching works
Rules are evaluated as described in RFC 9309, the standard Google follows: the group for the most specific matching user-agent applies (otherwise *), * matches any characters, $ anchors the end, and the longest matching rule wins — Allow wins a tie.
This is a simulation of the rules. It does not show how a search engine has actually crawled your site — use Search Console’s robots.txt report for that. robots.txt controls crawling, not indexing, and is not a security measure.
Related tools
robots.txt Generator
Build a clean robots.txt with sitemap lines, blocked folders and optional AI-crawler rules.
Runs in your browser
Website SEO Audit
Check one page for 35+ technical, on-page, indexability and security issues, plus its robots.txt and sitemap.
Live check
Meta Tag & Head Inspector
Fetch a URL and list its title, description, canonical, robots, Open Graph, Twitter, hreflang and headings.
Live check