Robots.txt Generator

Create a valid robots.txt file without touching a line of code. Choose which crawlers to allow or block, add disallow rules and your sitemap URL, and this free generator outputs a clean, ready-to-upload robots.txt for your site’s root.

How to use the Robots.txt Generator

1. Choose default allow or disallow for all bots.
2. Add specific user-agents and disallow/allow paths.
3. Add your sitemap URL.
4. Copy the file and upload it to your domain root.

Why robots.txt matters

Robots.txt tells search engine crawlers which parts of your site they can and cannot access, helping you protect private areas, save crawl budget, and avoid indexing junk pages. A single wrong rule can deindex your whole site, so generating a valid file is safer than editing by hand.

Rules worth knowing

• The file must live at your root: example.com/robots.txt.
• Disallow blocks crawling, not indexing. A blocked URL with external links can still appear in results, use noindex on the page instead when you want it out of the index.
• Never block CSS or JS that pages need to render; it breaks how Google sees your layout.
• Always declare your sitemap with an absolute URL.
• Paths are case sensitive and match by prefix.
• The most specific matching rule wins, and Allow can carve an exception out of a broader Disallow.

Common patterns

Most sites need very little: allow everything, block internal search results and cart or checkout paths, block faceted parameter URLs that create duplicates, and point to the sitemap. Staging environments should be protected by authentication rather than robots rules, since a disallow does not stop anyone from reading the site.

WordPress specifics

Leave /wp-admin/ disallowed but keep /wp-admin/admin-ajax.php allowed, since themes and plugins use it on the front end. Never block /wp-includes/, /wp-content/themes/ or /wp-content/plugins/, Google needs the CSS and JavaScript inside them to render your pages.

AI crawlers

Bots such as GPTBot, ClaudeBot, Google-Extended, CCBot and PerplexityBot honour robots.txt. Blocking them keeps your content out of model training and, in some cases, out of AI answers that could have cited you. There is a real trade-off: exclusion protects the content, inclusion can earn visibility and referrals. Decide deliberately rather than by default.

Crawl-delay and budget

Google ignores Crawl-delay; Bing and Yandex respect it. If crawling is genuinely straining your server, fix the underlying performance or use Search Console’s crawl rate settings rather than throttling every bot.

After you upload

Verify the file loads in a browser, then test key URLs in Search Console’s robots tester. A single stray Disallow: / left over from a launch is one of the most expensive one-line mistakes in SEO, and the meta tag analyzer will catch its on-page equivalent, a leftover noindex.

Key features

• Per-user-agent rules
• Allow/disallow paths
• Crawl-delay & sitemap directives
• Valid syntax
• Free

Frequently asked questions

Where does robots.txt go?

In your domain root, at yourdomain.com/robots.txt.

Does robots.txt stop indexing?

No, it blocks crawling; use noindex to prevent indexing.

Can I block a specific bot?

Yes, target it by user-agent.

Related free tools

Robots.txt Tester · Sitemap XML Generator · Google Index Checker. See all of our free SEO tools.

Want the links and rankings handled for you?

Our team runs the outreach, earns the placements and reports every live link. Explore our link building services or tell us your target pages for a plan.