Free tools / ai-crawler-checker
AI Crawler Checker
free · no signupTwelve AI user agents, one verdict each, with the exact robots.txt line that decided it — and the difference between blocking a training crawler and deleting yourself from ChatGPT’s search results.
Check a domain
This check runs on our server, and it has to. Your browser cannot read another
site’s robots.txt — the browser’s cross-origin rules forbid it. Any tool claiming to
check your domain’s robots.txt “privately in your browser” is either proxying it anyway
or not checking it.
we fetch /robots.txt and your homepage from our IP · we store no scan against your domain without you asking us to
robots.txt verdict
live check in buildThe live check is not deployed yet.
This page will not print a verdict about your site until the checker behind it is running. Nothing below is a guess about your domain, and an unrun check is never shown as a green tick — it is shown as not measured, with the reason.
The example underneath is what a completed result looks like. It is example data, on
example.com, and it is labelled as such.
| User agent | Operator | Verdict | Deciding rule |
|---|
not measured ≠ allowed, and never a zero — a failed fetch, a timeout or a field the checker did not return is an unrun check.
example result
example data · example.com · not a live check| User agent | Operator | Verdict | Deciding rule |
|---|---|---|---|
| GPTBot | OpenAI | blocked | Disallow: / — line 8, group User-agent: GPTBot |
| OAI-SearchBot | OpenAI | allowed | Allow: / — line 2, group User-agent: * |
| ChatGPT-User | OpenAI | allowed | Allow: / — line 2, group User-agent: * |
| ClaudeBot | Anthropic | blocked | Disallow: / — line 11, group User-agent: ClaudeBot |
| Claude-User | Anthropic | allowed | Allow: / — line 2, group User-agent: * |
| PerplexityBot | Perplexity | allowed | no rule — allowed by default |
| Perplexity-User | Perplexity | allowed | no rule — allowed by default |
| Google-Extended | blocked | Disallow: / — line 14, group User-agent: Google-Extended | |
| Applebot-Extended | Apple | allowed | no rule — allowed by default |
| Bytespider | ByteDance | blocked | Disallow: / — line 17, group User-agent: Bytespider |
| CCBot | Common Crawl | allowed | Allow: / — line 2, group User-agent: * |
| meta-externalagent | Meta | not measured | robots.txt fetch timed out on retry — the file was not read for this row |
a completed result also carries the second, separate check: your homepage requested from our server sending each bot’s user-agent string, with the HTTP status — sent from our IP, not theirs.
robots.txt — reference block
written from the agent list on this page · not from your siteWhy this tool exists
Blocking a training crawler and blocking a search crawler are two different decisions
A list of user agents with a row of green ticks and red crosses is close to useless without the one distinction that actually matters: some of these agents collect training data, and some of them fetch your page so an engine can answer a question and link to you. They are the same shape in a text file and opposite decisions in practice.
Block GPTBot and your content stops feeding OpenAI’s model training. ChatGPT can still find
you, cite you and send you a visitor. Block OAI-SearchBot and you have removed yourself from
ChatGPT’s search results. Plenty of sites did the first on purpose in 2023 and the second by accident
afterwards, with a wildcard.
This tool separates the four things a bot name can mean — because a single “AI bots: blocked” switch collapses all four into one, and three of them were decisions you never made.
The four things a bot name can mean
Same shape in the file. Different consequence when you disallow it.
01
A training crawler
Collects content that may be used to train a model. Blocking it is a licensing decision, not a visibility one.
02
A retrieval crawler
Builds the index an answer engine searches when it needs to cite something. Blocking it removes you from those answers.
03
A user-triggered fetch
Fires when a person asks the assistant about a specific page. Blocking it means someone pasting your URL gets nothing back.
04
A permission token that is not a crawler at all
Google-Extended and Applebot-Extended crawl nothing. They are opt-out switches on content already fetched. Disallowing them does not change search inclusion.
The agents this tool checks
| User agent | Operator | Type | What it does | What blocking it costs you |
|---|---|---|---|---|
| GPTBot | OpenAI | training | Crawls pages that may be used to train OpenAI models | Nothing in ChatGPT search. Training use only |
| OAI-SearchBot | OpenAI | retrieval | Crawls to surface and link sites in ChatGPT’s search results | ChatGPT search visibility. The expensive one to block by accident |
| ChatGPT-User | OpenAI | user fetch | Fetches one page when a user or tool asks for it | A user who pastes your URL into ChatGPT gets nothing back |
| ClaudeBot | Anthropic | training | Anthropic’s crawler; content may be used for model training | Training use |
| Claude-User | Anthropic | user fetch | Fetches a page in response to a Claude user’s request | The same paste-a-link case, on Claude |
| PerplexityBot | Perplexity | retrieval | Crawls to index pages Perplexity can cite in answers | Perplexity citations |
| Perplexity-User | Perplexity | user fetch | User-triggered fetch for one request. whether it obeys robots.txt: not verified against Perplexity’s published crawler docs | not verified — same source |
| Google-Extended | permission token | Not a crawler. Controls whether already-crawled content may be used for Gemini and Vertex AI grounding and training | No effect on Google Search ranking or inclusion | |
| Applebot-Extended | Apple | permission token | Not a crawler. Controls whether Applebot-crawled content trains Apple’s foundation models | No effect on Siri or Spotlight search inclusion |
| Bytespider | ByteDance | crawler | ByteDance’s crawler. stated purpose and robots.txt behaviour: not verified against ByteDance’s published documentation | not verified — same source |
| CCBot | Common Crawl | training | Crawls for the Common Crawl public archive, a known input to many training datasets | Indirect training use across many models |
| meta-externalagent | Meta | training | Crawls for Meta’s AI products and model training | Meta AI training use |
Operators add, rename and retire agents. Every row here comes from the operator’s own published documentation, and the cells marked not verified are the ones we have not re-confirmed — they say so rather than guessing. A stale list here is the single error that would discredit this tool, so it is re-checked on a schedule rather than when someone notices.
The precedence rule
Where accidental blocks actually come from
Almost never from someone typing Disallow: / under GPTBot. They come from the
precedence rule: a crawler obeys the most specific group that names it, and once it has one, it
ignores the User-agent: * group entirely. So a group added for one reason — a
crawl-delay line, a single blocked directory — quietly replaces every Allow rule you wrote
for everyone else. The bot is not reading both groups and merging them.
The second source is inheritance from a staging config, where Disallow: / under
* was correct and then shipped. Every AI agent obeys it, silently, and nothing in your
analytics tells you.
robots.txt is a request, not a lock
A well-behaved crawler reads the file and complies. Your CDN, WAF or bot-management rules operate a layer above it and simply return a 403 — and bot-management products now ship an “AI crawlers” category that can be switched on without anyone editing a robots file.
That is why this tool does a second, live check with the bot’s user-agent string. A green
robots.txt and a 403 at the edge is a real and common combination, and a robots-only checker
reports it as fine.
Limitations, stated plainly
We are not the bot. Our request carries the bot’s user-agent string from our own IP, so a firewall that filters by verified IP range may treat us differently than the real crawler — the live check catches obvious blocks, not every one.
We read robots.txt and HTTP status; we cannot see whether an engine has actually fetched your
pages, which lives in your server logs. And allowing every agent guarantees nothing about being mentioned
or cited — access is a precondition, not a cause.
Fair questions.
The same five answers feed this page’s own FAQPage markup — one array, one source.
Does blocking GPTBot remove me from ChatGPT?
No. GPTBot collects training data. Appearing in ChatGPT’s search results depends on
OAI-SearchBot, and pasting a link into a chat depends on ChatGPT-User. They are
three separate agents and three separate decisions. Blocking the first while allowing the other two is a
coherent position: no training use, still findable.
Should I block AI crawlers?
It depends on what your content is for, and this tool does not push you either way. If your traffic model is people reading your pages and you have no interest in being an answer source, blocking training crawlers is defensible. If you want to be cited in AI answers, blocking retrieval crawlers works directly against that. The only bad version is doing it by accident.
Why can’t this run in my browser?
Because a browser cannot fetch another site’s robots.txt — cross-origin rules block the
request. The check runs on our server. We say so on the tool itself, because “runs in your browser” is
only worth saying where it is true, and here it isn’t.
My robots.txt allows everything, so why aren’t AI engines citing my pages?
Access is one requirement of several. The page still has to render without JavaScript for agents that do not execute it, be reachable from your sitemap and internal links, and actually answer the question being asked. Crawler access removes a blocker; it does not create a mention.
Does Google-Extended affect my Google rankings?
No. Google-Extended is not a crawler and has no role in Search. It controls whether content
Googlebot already fetched can be used for Gemini and Vertex AI grounding and training. Disallowing it is a
decision about AI use, not an SEO decision.
Crawler access is the precondition, not the result.
See whether AI actually mentions your brand.