Free tools / ai-crawler-log-analyzer
FREE TOOL · NOTHING LEAVES THIS TAB
AI Crawler Log Analyzer
Your server writes down every visit it handles. Paste that file in and this reads it back to you: which AI crawlers actually arrived, on what dates, what they asked for, and what your server gave them. It is the one question a robots.txt check cannot answer — that file says who is allowed in, and this says who came.
Your access log
nothing loaded
Drop a log file here
The file is opened by your browser and read in this tab. It is not uploaded, and there is no address to upload it to — this page has no server behind it.
Plain text, or the current day’s file out of /var/log/nginx/, /var/log/apache2/ or whatever your host calls its logs directory. Compressed .gz archives are not read: unzip one first.
The example is fabricated — invented addresses in the documentation range, invented paths, invented dates. It exists to show you the shape of the output before you hand it anything real. It is not data about anyone.
What the log shows
waitingNothing read yet.
Paste a log on the left, or drop the file in, and press Read the log. The counts, dates and paths below will come out of the lines you gave — and the report will say, every time, exactly how many lines that was.
Runs entirely in your browser. Nothing you paste here is sent anywhere. (checkable — open your network tab, paste a log and press the button: no request fires. Copy and download are counted as clicks; the log is not.)
THE RULE THIS TOOL IS BUILT AROUND
A crawler missing from your log is not a crawler that never came
This is the whole reason a log analyzer is easy to get wrong. The output looks like a verdict — nine bots listed, three of them absent — and the absent three read as a finding. They are not one.
Absence in a log is evidence about the log. It becomes evidence about the crawler only once you know the log is complete, and almost no log you can paste into a browser is.
So this page will never print the words never crawled you. What it prints instead is not present in the lines you gave us, which is the same observation without the invented conclusion. Five ordinary things produce that observation, and none of them means what the shorter sentence would:
- 01The window is too short. A crawler that visits monthly leaves nothing at all in a log covering Tuesday. The report states the first and last timestamp it found, so you can see the window before you read anything into a gap.
- 02The log rotated. Most servers cut the file nightly or at a size limit and archive the rest. What is left in the live file is the tail, not the history.
- 03Something in front absorbed the request. A CDN or a security layer that serves from cache, or blocks at the edge, answers without your origin server ever hearing about it — so the visit happened and your log has no line for it. Cloudflare, Fastly and most managed hosts sit in exactly that position.
- 04You pasted one server’s log. Two application servers behind a load balancer write two files, and each holds roughly half of everything.
- 05Two of the twelve tokens never appear in any log, by design.
Google-ExtendedandApplebot-Extendedare opt-out switches, not crawlers — no request is ever sent under either name. Their absence is guaranteed and carries no information at all, which is why the report puts them in a section of their own rather than in the missing list.
The same rule in one line, and it is the rule the rest of this site runs on too: not measured is not zero.
WHAT IT READS
Two log formats, read properly — and everything else named, not guessed
A parser that half-understands a format silently produces half a report, and a half report about a crawler is worse than no report. So this one recognises a small number of shapes exactly, and refuses the rest out loud.
READ IN FULL
Combined
The default on Apache and on nginx, and the only common format that records the user-agent — which is the field every answer on this page depends on. address - - [date] "GET /path HTTP/1.1" 200 1234 "referrer" "user-agent"
READ IN FULL
Combined with a leading host
The same line with the site’s own name in front of the address, which is what shared hosting and cPanel write when one server answers for several sites. Detected by shape, not by a setting you have to know.
RECOGNISED, THEN REFUSED
Common
The older format, still the default in a few places. It parses cleanly and it is useless here: it has no user-agent field, so nothing in it can identify a crawler. The report says that rather than reporting twelve absences.
Anything else — a JSON access log, an IIS or W3C extended log, a CDN’s CSV export, or a file that is simply not a log — is reported as unrecognised, with the first line it could not read quoted back so you can see what it choked on. It is never guessed at. A format this page cannot parse is a format it says it cannot parse.
WHAT IT TELLS YOU
Four questions a log can answer, and it answers them from the lines
01
Who arrived
Each of the twelve tokens on our AI bot list, matched against the user-agent string on every line. One list, published on that page, read by this one.
02
When, and how often
First seen, last seen and the request count for each, plus the window the whole log covers — because a count with no window attached is a number without a unit.
03
What they asked for
How many distinct paths each one fetched and the ones it fetched most. A crawler that has only ever seen your homepage has only ever read your homepage.
04
What they got back
The status codes your server answered with, grouped. Every request refused, or a run of missing pages, is a fact about your side of the exchange and is worth knowing.
None of that is a prediction. It is a description of lines that already exist on your disk, and the report is written to be re-checkable: every number in it can be recovered from the log with grep and a bit of patience.
WHAT A LOG CANNOT TELL YOU
Two limits, and neither of them
gets softened here.
A visit is not a citation
A crawler fetching your pricing page tells you the page was fetched. It does not tell you that any assistant went on to name you, quote you, or link to you — those are separate events, and one does not follow from the other. Plenty of pages are crawled thoroughly and never surface in an answer.
So no number on this page is a visibility number. Whether an engine actually names you is a different measurement, made by asking the engines rather than by reading your disk, and it lives on the AI Visibility Checker.
A user-agent is a claim, not an identity
The user-agent string is written by whoever is making the request, and nothing stops any piece of software from sending GPTBot. Confirming that a request really came from the company whose name is on it means checking the source address against that operator’s published ranges or a reverse lookup — and this page does not do that, because it would mean sending your log somewhere, which is the one thing it is built not to do.
Read the output as: lines claiming to be this crawler. For the well-behaved operators that is the same thing. For anything pretending, it is not, and no log analyzer that runs in a browser tab can tell you which you have.
FAQ
Fair questions.
Five answers, written once and used twice — the version you are reading and the version a machine reads are the same text.
Is my log uploaded anywhere?
No. The file is opened and read by your own browser, and there is no address it could be sent to — this page has no server behind it and makes no request carrying anything you paste. That is not a courtesy, it is the only honest way to build it: an access log contains your visitors’ IP addresses, which is personal data about other people, and taking a copy of it would create an obligation we have no reason to take on. Open your network panel, paste a log, press the button, and watch nothing fire.
Where do I find my access log?
On a server you control, usually /var/log/nginx/access.log or /var/log/apache2/access.log. On shared hosting there is normally a raw access log download in the control panel, often under a name like Raw Access or Logs. On a fully managed platform there may be none you can reach at all, in which case this tool has nothing to work with and the honest answer is that the question cannot be answered from your side.
A crawler is missing. Does that mean it never visited?
No, and this page will not say it does. It reports that a token is not present in the lines you supplied, which is a fact about those lines. A short window, a rotated file, a CDN answering in front of your server, or one server out of several all produce the same absence for reasons that have nothing to do with the crawler. Two of the twelve tokens, Google-Extended and Applebot-Extended, are opt-out switches rather than crawlers and never appear in any log at all.
My log is enormous. What happens?
It reads as much as it safely can in a browser tab and then stops, and the report states how many lines it read and how many it did not. The answer is about the lines it read and says so. If the part it reached is the wrong part — the tail of today rather than the week you care about — cut the section you want with a text editor first, or filter it on the server before you bring it here.
Does being crawled mean an AI will mention me?
No, and anyone telling you otherwise is selling something. Being fetched is a precondition, not a result: a page has to be reachable before it can be read, and being read does not oblige any model to name you, quote you or link to you. This page reports what arrived at your server. Whether an engine names you is measured by asking the engines, which is a different tool and a different kind of number.
THE LOG IS THE EVIDENCE
Getting crawled is the floor,
not the finish.
Score your site against the technical checks AI engines depend on, and see what is standing between a fetch and an answer.