Free tools / ai-bot-list
REFERENCE PAGE · 12 TOKENS · NOTHING TO TYPE
AI Bot User-Agent List
The twelve AI crawlers and agents worth naming in a robots.txt, spelled exactly as their operators publish them, with who runs each one, what it is for, and the column most lists leave out: whether blocking it costs you a route into answers, or only the requests it makes.
Where these came from, and why that matters. Every token below is copied from the list our own crawler checker evaluates, which spells them as their operators publish them. The purpose column is that same table’s. A user-agent string recalled slightly wrong is worse than no list at all: somebody pastes it into a robots.txt, the rule never matches anything, and the file reads as a decision that was never enforced.
The fourth column is our reading of what blocking each one removes. Where an operator’s own documentation does not settle a behaviour, the row says undocumented rather than filling the gap with something plausible.
THE COLUMN THAT DECIDES THINGS
Four kinds of bot, and blocking costs something different in each
“Block the AI bots” is one instruction covering four unrelated decisions. Sorting them this way is what makes the list usable.
Answer route
Feeds an index that answers people now. Blocking it removes your pages as a source on that surface. This is the one where blocking can cost you a citation.
Fetch on request
Only visits when a person asks the assistant to open a page. Blocking closes that door, and only at the moment someone asks for you by name.
Training input
Feeds model training. Whether blocking changes how a future model describes you is not published by any operator and is not something we have measured. What blocking definitely saves is the requests.
Opt-out token
A preference about content the operator already holds, not a visitor. Blocking it costs no requests, because none are made under this name.
The honest version of “does blocking cost me visibility?”
For the answer routes and the on-request fetchers, yes, in a way you can reason about: you are removing your own site from a place an assistant looks. For the training crawlers, nobody outside the labs can tell you, because none of them publish the relationship between a training corpus and whether a model names a business. Anyone who quotes you a figure for that is quoting a guess.
So the sentence we will stand behind is narrow: blocking a training crawler saves you the requests, and its effect on whether you are named later is unknown and unmeasured — by us and by everybody else with a public method.
THE LIST
Twelve tokens, grouped by who operates them
Copy the token exactly, including capitals and hyphens. Matching is case-insensitive and prefix-based, which cuts both ways: Claude matches both Anthropic tokens at once, and a typo matches nothing at all.
OpenAI
GPTBot
OpenAI
Model training.
Blocking removes your pages from OpenAI’s training crawl and saves you those requests. Any further effect is unmeasured.
OAI-SearchBot
OpenAI
ChatGPT search index.
This is the one to think hardest about. Blocking it takes your pages out of the index ChatGPT search draws on, which is a live route into answers rather than a training question.
ChatGPT-User
OpenAI
User-triggered browsing.
Fires when somebody in ChatGPT asks it to open a page. Blocking it means the one person who went looking for you gets nothing back.
Anthropic
ClaudeBot
Anthropic
Crawling and training.
Recorded as crawling and training together.
undocumented: which half a block removes is not settled by the description we hold, so we do not split it for you.Claude-User
Anthropic
User-triggered browsing.
The same shape as ChatGPT-User: a fetch that happens because a person asked for it.
Perplexity
PerplexityBot
Perplexity
Search index.
Perplexity’s answers carry sources on the face of them. Blocking this removes your pages from what it can put there.
Perplexity-User
Perplexity
User-triggered browsing.
The on-demand half. Blocking it and allowing PerplexityBot are two different decisions, which is exactly why both tokens exist.
Google-Extended
Gemini and AI training opt-out.
Google-Extended never visits your site — it is a switch about AI use of content Google already has, with no part in ranking or listing. Blocking it cannot cost you a request, because none arrive under it.
Apple
Applebot-Extended
Apple
Apple AI training opt-out.
Recorded as an opt-out token rather than a crawler, the same shape as Google-Extended.
undocumented: we have not confirmed whether any request ever arrives carrying this exact token, so the badge follows the recorded purpose and not a measurement of ours.ByteDance
Bytespider
ByteDance
Model training.
Blocking saves you the requests. No public method connects this crawl to whether any assistant names you.
Common Crawl
CCBot
Common Crawl
An open crawl corpus that many models are built from.
One step removed from every other row here: it crawls once, and other people build on the result. Blocking it removes you from a public corpus rather than from one company’s index.
undocumented: which models use the corpus, and how, is not something we can enumerate — so the downstream effect of blocking it is unknown, not zero.Meta
meta-externalagent
Meta
Model training.
Lower case, with a hyphen, exactly as written. Blocking saves you the requests; the rest is unmeasured.
What this list is not
It is not complete, and it does not claim to be. These are the twelve tokens our crawler checker evaluates. New agents appear, operators rename them, and any list like this is behind the moment somebody ships a new one. Treat it as twelve you can act on today, not as every visitor that will ever reach your server.
It is also not enforcement. A user-agent string is a claim the visitor makes about itself, and nothing stops software from sending any string it likes. Everything on this page is about the well-behaved half.
THE MISTAKE THAT UNDOES THE WHOLE FILE
A crawler obeys exactly one group
This is the part that turns a careful robots.txt into an accident, and it is not obvious from reading the file.
What people expect
That a named group is added on top of the rules in User-agent: * — so naming GPTBot to give it one extra permission leaves everything else in place.
What actually happens
If any group names a crawler, the wildcard group is ignored entirely for that crawler. Every rule you meant it to inherit is gone. Where several groups match, the longest matching user-agent value wins.
So a named group has to repeat the rules you still want, in full. Naming a bot to tighten one thing is how people accidentally loosen everything else.
Our robots.txt generator writes complete groups for this reason, and the AI Crawler Checker reads your real file and reports which group each of these twelve actually lands in.
QUESTIONS
Before you paste anything into a live file
Where do I put these tokens?
In your robots.txt, at the very top of your site. Each token gets its own User-agent line and its own complete set of rules underneath it. A crawler obeys exactly one group and never merges a named group with the wildcard group, so a named group that leaves a rule out has silently switched that rule off for that crawler.
Does blocking these hurt my Google rankings?
Blocking the AI-specific ones does not affect Google Search. Google-Extended never visits your site, and has no part in ranking or listing. Blocking Googlebot itself is a different and far more serious act, and Googlebot is deliberately not on this list.
Is this list complete?
No, and it does not claim to be. These are the twelve tokens our crawler checker evaluates. New agents appear and operators rename them, so treat it as twelve you can act on rather than as every agent that will ever reach your server.
Will blocking stop AI from mentioning my business?
No. It stops the well-behaved ones from reading your pages. An assistant can still name you from anything it picked up elsewhere, including press coverage, directories, forums and a competitor’s comparison page. Blocking removes your site as a source while leaving everyone else’s description of you in place.
How do I know a bot actually obeyed?
Not from this page. A user-agent string is a claim the visitor makes about itself, and nothing stops anything from sending any string. What you can check is your own side: whether your robots.txt says what you meant it to say, and whether your server or CDN hands a crawler something different from what it hands a browser.
A LIST IS NOT A FILE
Read your real robots.txt
against all twelve.
The AI Crawler Checker fetches your live file and reports one verdict per token, with the exact rule line and the group that produced it. A row it cannot read comes back as not measured, never as allowed.