Robots.txt Tester

Check whether a specific URL is allowed or blocked for a given crawler, using real rule precedence. Runs entirely in your browser — nothing is uploaded, nothing is stored.

How precedence is decided

Crawlers do not read robots.txt top to bottom. They select the most specific user-agent group that matches them, then apply the longest matching rule — so Allow: /wp-admin/admin-ajax.php wins over Disallow: /wp-admin/. This tester replicates that logic, including * wildcards and $ end-anchors.

Blocking is not deindexing

A disallowed URL can still appear in search results if other pages link to it — Google simply cannot read it, so it shows a bare URL with no snippet. To remove a page from the index you need a noindex tag on a crawlable page, or the removal tool. Blocking a page you also want deindexed prevents Google seeing the noindex.

The mistakes that cost traffic

• Disallow: / left over from a staging environment.
• Blocking CSS or JS, so Google cannot render the page properly.
• Blocking a directory that contains money pages by accident.
• Assuming robots.txt protects private content — it is a public file and a roadmap for anyone curious.

Test before you deploy

Paste your proposed file here and test your top ten URLs plus a handful you intend to block. Then confirm the live file after deployment — robots.txt is one line away from removing a site from search.