Robots.txt Tester
Free robots.txt tester. Check whether a URL is allowed or blocked by your robots.txt, validate syntax, and catch crawl-blocking mistakes. Instant results, no signup.
Make sure your robots.txt does exactly what you intend. Enter your robots.txt (or your domain) and a test URL, and this free tester tells you whether that URL is allowed or blocked for a given crawler, and flags syntax errors before they cost you traffic.
Robots.txt tester
Check whether a specific URL path is allowed for a given crawler, using longest-match rule precedence.
How to use the Robots.txt Tester
- Paste your robots.txt or enter your domain.
- Enter a URL and choose a user-agent.
- Run the test.
- See whether the URL is allowed or blocked, and why.
How testing robots.txt keeps your pages crawlable
A misconfigured robots.txt is one of the fastest ways to accidentally block important pages or your entire site from Google. Testing specific URLs against your rules confirms crawlers can reach what should be indexed and are blocked from what should not, catching costly mistakes early.
How precedence is decided
Crawlers do not read robots.txt top to bottom. They select the most specific user-agent group that matches them, then apply the longest matching rule, so Allow: /wp-admin/admin-ajax.php wins over Disallow: /wp-admin/. This tester replicates that logic, including * wildcards and $ end-anchors.
Blocking is not deindexing
A disallowed URL can still appear in search results if other pages link to it, Google simply cannot read it, so it shows a bare URL with no snippet. To remove a page from the index you need a noindex tag on a crawlable page, or the removal tool. Blocking a page you also want deindexed prevents Google seeing the noindex.
The mistakes that cost traffic
Disallow: /left over from a staging environment.- Blocking CSS or JS, so Google cannot render the page properly.
- Blocking a directory that contains money pages by accident.
- Assuming robots.txt protects private content, it is a public file and a roadmap for anyone curious.
Test before you deploy
Paste your proposed file here and test your top ten URLs plus a handful you intend to block. Then confirm the live file after deployment, robots.txt is one line away from removing a site from search.
Key features
- Allow/block testing per URL
- User-agent selection
- Syntax validation
- Rule-match explanation
- Free
Frequently asked questions
Why is my page blocked?
A Disallow rule matches its path, the tester shows which rule.
Can robots.txt block Googlebot only?
Yes, rules can target specific user-agents.
Does a blocked page still appear in Google?
It can appear without a snippet if linked elsewhere, use noindex to remove it.
Related free tools
Robots.txt Generator · Google Index Checker · Sitemap XML Generator. See all of our free SEO tools.
Want the links and rankings handled for you?
Our team runs the outreach, earns the placements and reports every live link. Explore our link building services or tell us your target pages for a plan.
Other free tools
Found problems you would rather not fix yourself?
Send us the scan output. We will tell you which issues actually affect rankings, and quote the work if you want it handled.