Marketing · Free Tool
Robots.txt Generator
Build a robots.txt file with custom Allow/Disallow rules, your sitemap URL, and one-click toggles to block or allow AI crawlers like GPTBot and ClaudeBot.
About this tool
Build a robots.txt file the way you'd actually use one: set Disallow (and optional Allow) paths for all crawlers, add your sitemap URL, then use the dedicated AI crawler toggles to explicitly allow or block bots like OpenAI's GPTBot, Anthropic's ClaudeBot, Google-Extended, Common Crawl's CCBot, and PerplexityBot — each gets its own User-agent block, separate from your general rules.
This split matters because AI crawlers are increasingly treated differently from search-engine crawlers: a site might happily let Googlebot index everything while blocking GPTBot from using content for model training, or vice versa. The generator keeps each bot's rule explicit and readable rather than burying the decision in one shared block.
Common uses
- Block specific directories (like /admin/ or /cart/) from all crawlers while keeping the rest of the site indexable
- Explicitly allow Googlebot and Bingbot while blocking AI training crawlers like GPTBot and CCBot
- Declare your sitemap location so crawlers discover your full URL list immediately
Frequently asked questions
Does blocking GPTBot or ClaudeBot in robots.txt stop AI from ever seeing my content?
It stops those specific crawlers from bulk-scraping your site for training or indexing purposes. It doesn't prevent a user from pasting your content into an AI chat themselves, and it's a voluntary honor-system directive — well-behaved bots respect it, but nothing technically enforces it.
What's the difference between Disallow in robots.txt and a noindex tag?
Disallow tells a crawler not to fetch a URL at all, so it can't see the page's content — but the URL can still appear in search results without a description if other sites link to it. A noindex meta tag, by contrast, requires the crawler to fetch the page and explicitly tells it not to index that result. Use noindex if you want a page to never show in search results; use Disallow to save crawl budget or keep a crawler out entirely.
Do I need a robots.txt file at all?
Not strictly — if you omit one, most crawlers assume everything is allowed. But an explicit robots.txt with a Sitemap directive is still good practice: it removes ambiguity and gives crawlers a direct path to your full URL list.
Is my data uploaded to a server?
No. This tool runs entirely in your browser using JavaScript — nothing you type is transmitted, logged, or stored anywhere. You can even use it offline once the page has loaded.