How to read your server logs for AI crawlers

What your access log actually records

An access log is one line per request your server answered, written as it happened, with no script required at the other end.

That last part is why it matters here. A page-view counter is JavaScript: it runs in a browser, after the page loads. AI crawlers fetch the HTML and leave — no JavaScript runs, so no analytics event is ever recorded. A crawler can read every page you own and your dashboard will show a flat line.

The log has no such gap, because the web server writes the line itself. If a request arrived, there is a line. Most hosts keep these at /var/log/nginx/access.log or /var/log/apache2/access.log; on managed WordPress hosting they are usually a download in the control panel, often called “raw access logs”.

Which AI crawlers to look for

Nine user-agent tokens cover the assistants most people are asking about, and every one of them is published by the vendor that operates it.

One token you will never see, however hard you look: Google-Extended. It is a control you can write in robots.txt, not a crawler — no request ever arrives under that name. Googlebot does the fetching, and Google-Extended only governs what Google may do with content it already has.

The full AI bot list carries the rest, including the training-only crawlers from ByteDance, Common Crawl and Apple.

A log line, field by field

Both common log formats put the same facts in the same order, and five of those fields answer the question you came with.

An nginx or Apache “combined” line looks like this, wrapped here for reading:

Read it left to right. 203.0.113.42 is the IP that connected — the thing you verify, because it cannot be faked as easily as a name. The timestamp is when. GET /pricing/ is which page it asked for. 200 is what your server answered. 18422 is how many bytes it sent. The quoted text at the end is the user-agent: a self-declared label, and the only part of the line the visitor chooses.

That is the whole trick of log reading. The user-agent tells you what something claims to be; the IP tells you what it is.

The commands, if you have terminal access

Three commands answer the three questions people actually have.

Did any AI crawler reach me at all, and how often?

Which pages did one of them read? This prints the paths a single bot asked for, most-fetched first:

Is it getting what it asked for? This counts the status codes your server returned to that bot — the single most useful line in this guide:

A column of 200s is healthy. A column of 403s means something is turning the crawler away, and the next section of this guide is about finding out what.

Is it really the bot, or something wearing its name?

Anyone can put GPTBot in a user-agent string, so a name in the log proves nothing until the IP agrees with it.

Each operator publishes the addresses its crawlers use, which turns verification into a lookup:

• OpenAI publishes a separate list per bot: openai.com/gptbot.json, openai.com/searchbot.json and openai.com/chatgpt-user.json.
• Perplexity publishes perplexity.ai/perplexitybot.json.
• Anthropic publishes claude.com/crawling/bots.json, and documents the three bot names rather than a full user-agent string — so match on the token, then confirm the IP.
• Google publishes developers.google.com/search/apis/ipranges/googlebot.json, and also supports reverse DNS: the address should resolve to a googlebot.com or google.com host, and that hostname should resolve back to the same address.

A line whose name says one thing and whose IP is nowhere on the vendor’s list is not that crawler. It is a scraper, a monitoring service, or someone testing your site.

What a crawl proves, and what it does not

A log line proves your page was fetched. It says nothing about whether an assistant ever used it in an answer.

This is the conclusion people reach by accident, and it is worth being blunt about. A crawler reading your pricing page is a necessary condition for being cited from it, not evidence that you were. The fetch and the answer are separate events, and the log only witnesses the first one. Being read is also not being named: an assistant can name your business from training data without fetching anything, and can fetch your page and still recommend a competitor.

What the log genuinely settles is the question that comes first: is my site even reachable by the thing that writes these answers? If the answer is no, nothing downstream can work. If the answer is yes, you have ruled out the dullest explanation and can go and measure the answers themselves — which is a different job, done by asking the assistants the questions your customers ask and recording what comes back.

When a bot never appears

An empty result usually has one of four causes, and they are quick to separate.

1. Your robots.txt says no. The honest first check, because it is the one you control and the one most often wrong by accident. Our AI crawler checker reads your live file and reports each bot’s state; it is also the file where a blanket Disallow aimed at scrapers catches the search crawlers you wanted.
2. Something in front of your server is refusing them. A CDN bot rule or a firewall can answer 403 before your application sees the request. If your log shows the request and a 403, that is your stack; if it shows nothing at all, look at the edge, where the refusal is logged separately or not at all.
3. Nobody has asked for your page yet. New sites, and pages nothing links to, go uncrawled for a long time. That is not a fault to fix in the log; it is a discovery problem, solved by internal links and sitemaps.
4. You are reading the wrong log. Rotated files (access.log.1, access.log.2.gz) often hold the window you care about. Search the archive before concluding silence.

The status codes that matter in an AI crawler’s lines

Five codes carry nearly all the meaning, and each one points at a different owner.

• 200 — sent. Check the byte count too: a 200 that returned 400 bytes delivered an empty shell, which is what a JavaScript-rendered page looks like to something that runs none. The render check shows that directly.
• 301 / 302 — redirected. Fine once, wasteful in chains, and a loop means the crawler never gets the page.
• 403 — refused. Your edge or your server decided against it. This is the one to chase.
• 404 — the URL is gone. If a crawler keeps asking, something still links to it.
• 429 or 5xx — you ran out of capacity, or the application broke. Anthropic documents support for the non-standard Crawl-delay directive in robots.txt, which is the polite lever when a crawler is simply arriving too fast.

What the pattern of fetches tells you

One line is a fact; a month of lines is a shape, and the shape is more useful than any single request.

Which pages they keep coming back to is the most practical signal in the file. Crawlers re-fetch what they treat as worth re-reading, so a page pulled weekly is being treated differently from one fetched once in March. If the pages you sell from are missing from that list while your blog archive is not, the gap is a brief: the crawler found a route to one and not the other.

Which bot is doing the fetching changes what the visit means. A ChatGPT-User or Perplexity-User line is a person’s question in motion — somebody asked something and the assistant went to your page during that conversation. A GPTBot or ClaudeBot line is collection, documented by both vendors as feeding model training rather than answering anybody right now. Same log, two different stories.

And the dates matter more than the totals. A crawler that visited heavily in August and not since has usually hit something: a block that appeared at your edge, a redirect introduced in a redesign, or a run of errors. A total for the quarter hides that; a count per week shows it in one line.

A five-minute routine, monthly

Checking logs is worth doing on a schedule and not worth doing daily.

Crawl behaviour changes over weeks, not hours, and a daily look produces noise you will learn to ignore. Once a month, answer four questions and write the answers down somewhere you will find them next month:

1. Did each bot you care about arrive at all? One count per token. Zero is the only number that needs acting on.
2. What did your server answer? The status-code column from the command above. Anything that is not mostly 200 is the month’s job.
3. Which pages did they take? Compare against the pages you actually want read. Mismatch is a linking problem, not a crawler problem.
4. Did anything stop? A bot that was arriving and no longer is, is the finding most people miss, because nothing breaks visibly when it happens.

Four numbers a month is a record. After three months it is a trend, and a trend is the only thing that will tell you whether a change you made did anything.

Reading the log without a terminal

If you cannot run shell commands, download the raw access log from your host’s control panel and read it in your browser.

Our AI crawler log analyzer does exactly what the commands above do — which bots arrived, which pages they asked for, what your server answered, and when — and it parses the file inside your own tab. The log never leaves your machine, and there is no endpoint to upload it to, deliberately: a server access log contains the IP addresses of everyone who visited your site, and accepting that onto our server would make us a custodian of your visitors’ data.

FAQs