Over three days in September, AI crawlers made 4,058 requests to my site. In the same three days they sent 7 readers back.
Googlebot and the other search crawlers made 3,101 requests over those days and sent 287 readers. Roughly eleven pages taken for every person delivered, against 580.
Both halves of that comparison come from the same three dates, which matters more than it sounds. My first attempt at these numbers compared two days of crawler data against thirty days of referrals and produced a ratio that was off by a factor of twenty.
Search Was Always a Trade
A crawler reads your pages, indexes them, and people arrive through the index. You pay in bandwidth and server load, and you get an audience. I never had to enjoy that bargain for the arithmetic to work, and on my site it still runs at about eleven pages taken per reader sent.
The assistants read more than Googlebot does and return almost nothing. That isn’t theft, and I’m not interested in arguing that it is. It’s a different transaction. What bothers me is how few publishers can see either side of it.
Your Analytics Plugin Cannot See Them
Almost every WordPress analytics tool counts a visit when a JavaScript beacon fires in the reader’s browser. That design is what lets these plugins work without cookies and without slowing your pages down, and it’s the right call for measuring people.
Crawlers don’t run JavaScript. So the entire category is invisible to the tool you would naturally open to check.
Google Analytics has the same blind spot for the same reason, and more of it. GA4 is a JavaScript tag. A crawler fetches your HTML, never executes the tag, and never appears in any GA4 report. If you have been reassuring yourself that GA doesn’t show much bot traffic, that is because it cannot show any. That is a separate question from whether GA4 suits a content site at all.
Server logs have the opposite problem. They record every request, which is why a log analyzer always reports numbers far above your analytics dashboard. They also count every crawler, uptime monitor and scanner as a visit, so the totals mean something quite different from what most people assume when they read them.
Three vantage points, three answers, and none of them is simply the truth.
What Is Actually Reading Your Site
Here is one day from my access log, counting page requests and excluding robots.txt:
- ChatGPT-User: 348
- meta-externalagent: 294
- PerplexityBot: 173
- ClaudeBot: 155
- Claude-User: 150
- Amazonbot: 128
- Bytespider: 127
- OAI-SearchBot: 93
- DuckAssistBot: 65
- GPTBot: 21
Worth knowing that these do different jobs. GPTBot and ClaudeBot gather training data. Search crawlers publish what they are, and so do these: ChatGPT-User and Claude-User fetch a page because someone asked a question and the assistant went to look, which is the closest thing to a referral that never becomes one. OAI-SearchBot and PerplexityBot build search indexes. Amazonbot and Bytespider belong to Amazon and ByteDance, and both companies describe them in terms of improving their own products.
Lumping them together as “AI crawlers” flattens all of that, which is what I have done above and what the total in any dashboard does too. Worth holding in mind before the number turns into an argument.
I Got This Wrong in My Own Plugin
I build Vireo Analytics, so I had a bot filter and a dashboard panel breaking crawler traffic into categories. I have compared it against the other privacy-first options separately. It told me AI crawlers were outnumbering search crawlers 22 to 1.
The server log for those same days said 1.3 to 1.
The filter was only recording crawlers that reached the tracking endpoint, which is to say crawlers that execute JavaScript. A handful do. The ones that render pages happen to be AI agents, so the category that could be seen was AI and the category that could not be seen was search. The result wasn’t merely low, it was bent, and bent in the direction that made the more alarming number look correct.
That is the failure mode worth taking from this. A measurement that is wrong in a flattering direction survives much longer than one that is obviously broken, because nothing about it prompts you to check.
The fix was to count crawlers when they ask WordPress for a page rather than when they run a script they will never run. That shipped this week, and if you are moving off GA there is a way to bring your history across rather than starting at zero.
Why Plugin Numbers Are a Floor
A plugin can only count requests that reach PHP. If your site runs full-page caching, and most do, a good share of crawler traffic is answered by nginx or Varnish before WordPress wakes up. Nothing running inside WordPress can see those requests, including my plugin.
So the crawler numbers in any WordPress plugin are a floor, not a total. The dashboard now says so rather than implying otherwise, which was the second thing I had to fix.
If you want the real figure, the access log is the only complete record. Most managed hosts expose it, and counting is a one-line job:
grep -icE "GPTBot|ChatGPT-User|ClaudeBot|PerplexityBot|Bytespider|Amazonbot|meta-externalagent" access.logRun the same thing against Googlebot|bingbot and compare the two. Then look at how many readers each actually sent you.
What I’d Do With the Answer
Not block them, in my case. The assistants that fetch a page because someone asked a question are the closest thing to a recommendation there is, and cutting those off to save bandwidth would be trading a small cost for a smaller future.
What the numbers changed is what I count as traffic. A dashboard that folds crawler requests into pageviews is telling you a story about demand that isn’t true, and every decision you make about what to write next rests on it.
Measure the two separately. Then decide.
Related
Leave a Reply