Free tools / robots-txt-generator
robots.txt Generator
free · runs in your browserA form on the left, the finished file on the right, updating as you type. Every bot you name gets a complete group — because a crawler obeys exactly one group and never merges it with *.
Choices
1 group
Loaded as an example, not a recommendation. Each preset is a decision about how your content may be used — see the four classes below the tool.
AI crawlers — tick to block
0 blocked
robots.txt
ok ✓Save it as robots.txt at the root of the domain — https://example.com/robots.txt, nowhere else.
Correctness pane
Runs entirely in your browser. Nothing you type here is sent anywhere. (checkable — open your network tab)
Why this tool exists
The mistake that makes a robots.txt do the opposite of what you meant
robots.txt looks like a config file and behaves like one only until you add a second group. Then this rule applies, and it is the source of most accidental blocks on the web:
A crawler obeys exactly one group — the most specific one that names it — and ignores every other group in the file, including User-agent: *.
Groups do not merge. So when a file ends up looking like this:
User-agent: * Disallow: /admin/ Disallow: /cart/ Allow: / User-agent: GPTBot Crawl-delay: 10
…you have not added a crawl delay for GPTBot. You have created a group for GPTBot that contains no Disallow at all, which means GPTBot may now crawl /admin/ and /cart/ — the rules you wrote for everyone no longer apply to it. (Google ignores Crawl-delay outright, so the line that motivated the whole group does nothing either.)
This generator writes complete groups. Name a bot and it inherits every path rule you set for *, written out explicitly in its own group, plus whatever you changed for it. What you read is what every crawler reads.
The rules the output follows
Standardised in RFC 9309 and implemented by the major crawlers:
01
Longest match wins
Between Disallow: /blog/ and Allow: /blog/public/, the longer matching rule decides. On an exact tie, Allow wins.
02
Disallow: with nothing after it means allow everything
Disallow: / means block everything. One character apart, opposite outcomes.
03
* and $ are supported
* matches any run of characters, $ anchors to the end of the URL. Disallow: /*.pdf$ blocks PDFs.
04
Location is fixed
https://example.com/robots.txt governs https://example.com only. Not the http:// version, not a subdomain, not another port. Each needs its own file.
05
Response codes matter more than people expect
A 404 means crawl everything. A persistent 5xx is treated by Google as disallow everything — a robots.txt that errors during an outage can pull your whole site out of crawling.
06
Google parses the first 500 KiB
Rules past that are ignored, which only bites hand-maintained monsters.
Choosing AI presets is a values decision, not an SEO trick
There is no configuration here that makes you more visible in AI answers, and this page will not pretend there is. What you are choosing is which uses of your content you consent to:
training
Training crawlers
GPTBot, ClaudeBot, CCBot, Bytespider, meta-externalagent take content that may end up in a model’s weights. Blocking them costs you nothing in search or citations.
retrieval
Retrieval crawlers
OAI-SearchBot, PerplexityBot build the index an engine cites from. Blocking them removes you from those answers.
user-triggered
User-triggered agents
ChatGPT-User, Claude-User, Perplexity-User fetch a page because a person asked about it. Blocking them breaks that for your own prospects.
opt-out tokens
Google-Extended and Applebot-Extended
Not crawlers at all — they are opt-out tokens for AI use of content already crawled, and disallowing them has no effect on Google or Apple search inclusion.
Pick deliberately. The failure this tool exists to prevent is picking by accident.
Blocking a URL is not removing it
robots.txt stops the fetch, not the listing. A blocked URL can still appear in a search index — discovered from links elsewhere, listed without a description, because the crawler was never allowed in to read the noindex you put on the page. To remove a page, allow crawling and serve noindex. Blocking it in robots.txt guarantees the opposite of what most people intend.
The same logic applies to AI engines: a model that learned your brand from third-party pages can name you whether or not it can crawl you. Blocking a crawler removes your site as a source; it does not remove your brand from a model’s memory.
Limitations, stated plainly
robots.txt is a request, not access control. Well-behaved crawlers comply; anything else ignores it, and a determined scraper reads the file as a map of what you consider sensitive — so never list a private path here. This tool writes the file; it cannot deploy it, and it cannot see whether your CDN or firewall is separately blocking bots at the edge (that is the crawler checker’s job). And no robots.txt configuration causes a mention or a citation in any AI engine.
Fair questions.
The same five answers feed this page’s own FAQPage markup — one array, one source.
Where does robots.txt go?
At the root of each origin: https://example.com/robots.txt. It governs that scheme, host and port only — http:// and https://, example.com and shop.example.com, each need their own file. A robots.txt in a subfolder is ignored entirely.
Does blocking AI crawlers hurt my Google rankings?
Blocking the AI-specific agents does not affect Google Search. Google-Extended is not a crawler and has no role in ranking or indexing; it only controls AI use of content Googlebot already has. Blocking Googlebot itself is a different and much more consequential action, which this tool will warn you about before you copy the file.
Will robots.txt stop AI models from mentioning my brand?
No. It stops compliant crawlers from fetching your pages. A model can still name your brand from anything it learned elsewhere — press coverage, directories, forums, competitors’ comparison pages. Blocking crawlers removes your site as a source while leaving everyone else’s description of you in place.
Is anything I type here uploaded?
No. This tool builds text from your selections in the page itself. There is no server call, no site fetch and no storage — checkable in your browser’s network tab. Our crawler checker is the opposite case and says so on its own page, because it has to read a live file from our server.
Can I just block everything and add exceptions later?
You can, and it is the most expensive reversible mistake in this file. Disallow: / under User-agent: * blocks every compliant crawler, and recovery is not instant — crawlers have to return, re-read and re-crawl before anything comes back. If you are unsure, block the specific agents you have decided about and leave the rest allowed.
The file is five minutes.
Knowing whether it changed anything is the hard part. See whether AI mentions your brand.