Who actually crawls you

What we are solving

Every guide tells you to publish llms.txt. None of them shows a log line where an agent asked for it.

So I read my own. One box, several small sites, per-site nginx access logs dated 19 July to 12 August 2026.

The window has a hole in it: nothing between 20 and 28 July survived rotation, so what follows is 16 days of records.

This is one operator’s traffic, not a sample of the web. It is still a measurement. The advice around it does not have even that.

The first thing the measurement found was me. My own address made 4,221 requests to this box in the window, and 81 of those went out under a crawler’s name.

Steps

Read the whole period first, and a single agent after that.

The Mine column is the rows whose first log field matched my own outbound address. I got that address with curl -s https://api.ipify.org from the same machine I had been poking the site from, and after that it is just subtraction:

zcat -f /var/log/nginx/example.access.log* \
  | awk 'tolower($0) ~ /claudebot/ {mine += ($1 == "203.0.113.9"); all++}
         END {print "logged", all, "| mine", mine, "| rest", all - mine}'

The test is crude. It catches one address only, so anything you did from a phone or from CI never lands in Mine. It also invents nothing, and it was enough to take two rows in the table below down to zero.

Agent	Logged	Mine	Rest
ChatGPT-User	341	4	337
YandexBot	244	3	241
bingbot	155	3	152
OAI-SearchBot	82	4	78
Googlebot	79	4	75
GPTBot	55	14	41
meta-externalagent	41	3	38
Claude-User	34	12	22
PerplexityBot	25	7	18
Amazonbot	22	2	20
Applebot	7	3	4
ClaudeBot	6	6	0
Perplexity-User	3	3	0

Two rows turned out to be entirely mine: every ClaudeBot and Perplexity-User request came from my address, so subtracting it leaves zero.

An earlier version of this page blamed those rows on an unknown scanner: it said one client had sent 56 different crawler names from a single address.

The count was close. The client was me. That address sent 56 distinct User-Agent strings in the window, and 18 of them were crawler names, all from curl runs testing my own access rules.

Google-Extended was in that set. It has no user agent of its own. It is a control token you write in robots.txt, and Google’s ordinary agents do the crawling.

So the string was real and no crawler stood behind it, and you will never grep Google-Extended out of your own log, not unless you type it at yourself.

The largest row is the one I cannot attribute

ChatGPT-User is top of the table, and it is the row the log explains worst, because that agent fires when a person pastes a URL into a chat.

Four of the 341 came from my own address, and that was curl with the string set by hand. The other 337 came from 272 addresses. I never checked those addresses against a vendor’s published range.

If the agent is genuine, the fetch leaves OpenAI’s servers whether I pasted the link or a stranger did. The address would not tell us apart either way.

What the log can say is thin. Requests arrive around the clock, and that is easier to see than to read — here is the same agent on a fresh reading taken on 15 August.

ChatGPT-User by hour of day, 317 requests over 15 days

00:0012:0023:00

Всего 317. Наведите на столбик, чтобы увидеть значение.

ChatGPT-User by hour of day, 317 requests over 15 days. 00:00 — 5, 01:00 — 9, 02:00 — 4, 03:00 — 10, 04:00 — 8, 05:00 — 8, 06:00 — 17, 07:00 — 9, 08:00 — 23, 09:00 — 16, 10:00 — 14, 11:00 — 12, 12:00 — 17, 13:00 — 20, 14:00 — 16, 15:00 — 15, 16:00 — 18, 17:00 — 18, 18:00 — 21, 19:00 — 13, 20:00 — 15, 21:00 — 14, 22:00 — 9, 23:00 — 6.

Read on 15 August 2026 from the voice-ai log. Not one hour is empty, and the dip falls at night in UTC.

Of the 337 requests on 12 August, thirty-eight fell within five minutes of a request from my own address.

Eleven per cent is weak evidence against “it was all me”, and about who did come, it says nothing at all. Separating those two stories needs a window in which I do not touch the site. I have not run one.

Read the whole ChatGPT-User row that way. It is a count of fetches by an agent a human has to trigger, and the log does not say who that human was.

What they asked for

The AI agents, taking ChatGPT-User, OAI-SearchBot, GPTBot, PerplexityBot, Perplexity-User, Claude-User, ClaudeBot and meta-externalagent together, with my own address removed:

Path	Requests
/	280
A blog post comparing speech-to-text models	64
/robots.txt	56
A blog post on transcribing a Telegram voice message	18
/ru	17
/sitemap.xml	14
The comparison post in Russian	6
The transcription post in Russian	2
/llms.txt	0

Forty-five fetches of llms.txt, forty-four of them mine

llms.txt took 45 requests in the period from all clients together. 44 came from my own address.

The one that did not was Semrush’s SiteAuditBot, on 11 August, and it got a 200. No AI agent asked for the file at all.

An earlier version of this page reported one fetch per agent and called the file a listing they glance at, and that reading turned out to be a single loop of mine.

Seventeen of those lines land between 18:21:12 and 18:21:15 on 7 August, one line per crawler name, all of them from my address. That is curl walking a list of user agents. Thirteen vendors did not arrive in four seconds.

So llms.txt is not a route agents travel, and on this box in this window it was not a glance either. I kept the direction of the conclusion and dropped the number.

The procedure

1. Log per site, not per box — one access_log line inside each server block.

   server {
       server_name example.com;
       access_log /var/log/nginx/example.access.log;
   }

   A single global log without $host in its log_format cannot say which site a request hit.

2. Open the rotated archives, not only today’s file — zcat -f reads plain and gzipped alike.

   zcat -f /var/log/nginx/example.access.log* | wc -l

   Read the logrotate config for how far back the box keeps them. That is your real window.

3. Find your own address before you count anything — one command, and it changes every table below.

   curl -s https://api.ipify.org

   Mine was in the log 4,221 times, under 56 different user agents, and I read that as traffic for two weeks.

4. Subtract your address, not only your curl — a browser and a spoofed agent land in the same file.

   zcat -f /var/log/nginx/example.access.log* | awk '$1 != "203.0.113.9"'

   Filtering curl caught 26 of the 44 llms.txt lines that were mine, and the other 18 were my own crawler names and my own browser.

5. Count the agents over the whole window — one match per request, case-insensitive.

   zcat -f /var/log/nginx/example.access.log* \
     | awk 'match(tolower($0), /chatgpt-user|oai-searchbot|gptbot|perplexitybot|perplexity-user|claude-user|claudebot|meta-externalagent|googlebot|bingbot|yandexbot|applebot|amazonbot/) {
         print substr(tolower($0), RSTART, RLENGTH)
       }' \
     | sort | uniq -c | sort -rn

   The awk matters here, not the grep -oE this page once shipped. grep prints occurrences instead of lines, and a single bingbot request names itself twice: in the token and in the +http://…/bingbot.htm URL.

In this window that is 336 occurrences against 155 requests. awk takes the first match per line, so one line means one request again. 6. Count the paths, AI agents only, your address excluded — field seven is the request path in the combined format.

zcat -f /var/log/nginx/example.access.log* \
  | grep -iE 'ChatGPT-User|OAI-SearchBot|GPTBot|Perplexity|Claude-User|ClaudeBot|meta-externalagent' \
  | awk '$1 != "203.0.113.9" {print $7}' | sort | uniq -c | sort -rn | head -20

Here grep without -o matches whole lines, so this one counts requests already.

What these rows do not prove

The comparison I came for is 0 fetches of llms.txt against 56 of robots.txt. Every AI agent in the table took robots.txt and none took llms.txt. That is narrower than calling the file useless, and it is what this log supports: one box, 16 days.

/ took 280 requests, more than every other path combined, and that is not a preference for the home page. A bare domain pasted into a chat and an agent starting at the root produce the same line: the log records the path, never the reason. Keeping good pages reachable from the root is sound advice, but not from this table.

The comparison post took 64 requests, plus 6 in Russian — more than the rest of the blog together, and the whole of what nginx knows about it. A 200 means bytes went out. Whether a word of them reached an answer shows up in referrals from assistant hosts and in the answers themselves, neither of which is a log line.

robots.txt took 56 requests and sitemap.xml took 14. Those two rows are protocol, not interest in me: a crawler takes robots.txt at the start of every session, so a high count means the handshake works. Nobody writes posts about a correct sitemap, and agents still ask for it before any prose.

All of this rests on identities I never verified, and the window showed the cost twice: I read my own spoofed strings as thirteen vendors, then read Google-Extended off a line that cannot exist. Doing it properly means reverse DNS on each client address, or matching the ranges the vendor publishes.

What did not work

• Counting my own traffic as an agent’s. My address is in this log 4,221 times, 81 of them under a crawler name, and I published the totals with all of it inside. I blamed the strange rows on an unknown scanner. It was my own curl loop, and one call to api.ipify.org said so.
• The llms.txt finding, first time round. This page reported about one fetch per agent and built a conclusion on that shape. Seventeen of those lines are four seconds of a loop I ran on 7 August, and the real count over 16 days is 0.
• Reading the live log and stopping there. My first pass used a file that started that morning and showed 0 requests to llms.txt — right by accident, wrong by reasoning. The archives held 45, and 44 of those were mine.
• Publishing a command that did not produce the table. The grep -oE version printed here was case-sensitive and never matched bingbot; made case-insensitive it counted occurrences instead of requests and doubled that row. A reader running it got neither the table nor an error.
• Counting calendar days instead of days on disk. This page first said 24 days between the same two dates. logrotate keeps 14 dailies here, and nine of those days had already gone when I counted them.
• Attributing the totals to a single site. These counts come from every per-site log on the box, and 340 of the 341 ChatGPT-User requests belong to one site. The split is inferred from paths, not read from a field.
• Taking a name in the log as an identity. Every agent above is an unverified User-Agent string, and mine were plainly fake. Which of the rest are somebody else’s scanner, I still cannot say.
• Counting nginx as the total. A request that Cloudflare served from cache never reaches the box. These figures are a floor, not a total.
• Pulling the Cloudflare-side numbers for comparison. The API call failed on token permissions, so the comparison is not in this page. Whatever the edge absorbed is missing from every table above.

Verify

• Print your own public address and grep the log for it before you read a single total. Mine was the largest client on the box.
• Re-run the count next month and compare the shape rather than the totals. One window will not tell you what is normal on your own box.
• Check logrotate retention before you believe your window. Mine reached further back than the file I first opened.
• Request /llms.txt yourself, then find that line in the log. If it is missing, you are reading the wrong file.
• Count status codes per agent. A wall of 403 means somebody is refusing them. In the totals that reads exactly like nobody having come.
• Count lines, not occurrences. Pipe the same data through wc -l and through grep -o, and compare the two totals.
• Resolve the addresses of your top agents by reverse DNS, and match them against the vendor’s published ranges.
• Take a window in which you do not touch the site at all. It is the only way to read a user-triggered agent honestly. I have not taken one yet.
• Compare the log against your CDN’s own analytics. On Cloudflare that is AI Crawl Control, in the Crawlers and Metrics tabs. Everything the edge cached or refused is invisible to nginx.

Who owns which agent, and where a block actually lives: AI crawlers and llms.txt. What a fetch is still not: why AI answers cite someone else.