caprail

Analytics for AI agents: the complete guide

Maximilian Leodolteragentsanalytics

Bots make up roughly 51 percent of all web traffic, and AI crawlers are now about 22 percent of that bot activity (Cloudflare Radar, 2026). The analytics you already run was built for browsers, so the fastest growing visitors on your site are invisible by design. This guide explains what AI agent analytics is, the three kinds of AI traffic, why the server is the only place to measure all of it, and how to start.

Key takeaways

  • Bots are about 51 percent of web traffic and AI crawlers about 22 percent of bots (Cloudflare Radar, 2026), but script-tag analytics only counts humans who run JavaScript.
  • There are three distinct things to measure: crawlers, live fetchers, and the human visits AI assistants refer back to you.
  • AI referrals are small in volume but high in value. One study saw ChatGPT visits convert at 15.9 percent against 1.76 percent for search.
  • Server-side classification is the only method that catches headless agents, because they never run a browser.

What is AI agent analytics?

AI agent analytics is the measurement of non-human, AI-driven visits to your site, plus the human visits that AI assistants send back. It covers crawlers, live fetchers, agentic browsers, and AI referrals, and it captures them on the server, before any browser script could run. The category is new because the traffic is new.

Think of it as a second audience sitting alongside your human one. When someone asks an assistant about your product, that assistant may have read your pages hours earlier through a crawler, then pulled a page live to answer, then sent the person to you with a link. Three separate events, and your current dashboard records at most the last one. Agent analytics records all three.

Why does this matter now and not two years ago? Because the volume crossed a line. When agents were a rounding error, ignoring them was fine. At a fifth of bot traffic and climbing, they are part of your funnel whether you measure them or not.

Why can't traditional analytics see AI traffic?

It cannot see most AI traffic because it runs in the visitor's browser. Script-tag analytics fires only when a real browser downloads your page and executes JavaScript. As of 2026, Cloudflare Radar reports that 57.5 percent of HTML requests come from bots rather than humans, and almost none of those bots run your script. To the analytics you already have, those visits never happened.

There are two separate blind spots here, and it helps to name them. The first is the headless crawler. When a training or indexing bot reads your docs, no browser exists, so no script runs. The second is the referrer-less click. When someone taps a link inside a native assistant app, the referrer is often stripped, so the visit lands in your "direct" bucket with no hint that an assistant sent it.

The result is structural, not a tracking gremlin you can fix with one more tag. You can add every browser-side tool on the market and still see zero of the headless half. The only place that observes every request, human or agent, is the server. For more on why this is a foundation and not a feature, see the best new analytics tool.

Crawlers, fetchers, referrals: the three kinds of AI traffic

AI traffic is not one thing, and lumping it together hides the story. There are crawlers that read in bulk, fetchers that read one page to answer a live question, agentic browsers that act on a person's behalf, search bots that index, and separately the human referrals that assistants send you. Each behaves differently and means something different for your site.

Here is the practical split:

Caprail classifies every request into one of these on the server. A crawler reading you a thousand times is a very different signal from one fetcher pulling a page to close a sale, and you want to tell them apart.

How fast is AI traffic growing?

It is growing far faster than human traffic. In 2025, one benchmark found AI bot traffic rose 187 percent while human traffic grew just 3.1 percent (HUMAN Security, 2026). Within that, live fetchers grew fastest, with ChatGPT-User up several thousand percent year over year.

Traffic typeChange during 2025
AI bot traffic+187%
Human traffic+3.1%

Source: HUMAN Security, 2026 State of AI Traffic. Figures shift quarter to quarter, so treat any single number as a snapshot rather than a constant.

A word of caution on the headline figures you will see quoted elsewhere. The widely shared prediction that search volume will fall 25 percent by 2026 (Gartner, 2024) is contested, and several analysts argue the methodology overstates the shift. The safer read is directional. AI traffic is rising quickly, even if the exact slope is up for debate.

Does AI traffic actually convert?

In the studies published so far, yes, and at a rate that is hard to ignore. On one B2B site, ChatGPT referrals converted at 15.9 percent and Perplexity at 10.5 percent, against 1.76 percent for search organic (Seer Interactive, 2025). The volume was small, a little over a thousand AI conversions, so read it as one site's result rather than a universal law.

SourceConversion rate
ChatGPT15.9%
Perplexity10.5%
Search organic1.76%

Source: Seer Interactive, 2025, single B2B site. Treat as directional, not a benchmark.

Other measurements point the same way. Ahrefs reported that AI search drove 0.5 percent of visits but 12.1 percent of signups on its own analytics, roughly a twenty-fold edge per visit (Ahrefs, 2025). Adobe found AI-referred shoppers convert about 42 percent better and spend more per visit (Adobe, 2026). The pattern across independent sources is consistent: AI traffic is small but unusually qualified. To measure it on your own site, see how the markdown twin works.

How much are you being scraped for every visitor?

A lot more than most owners expect. Network-wide figures derived from Cloudflare Radar suggest some crawlers read thousands of pages for every human they send back. ClaudeBot has been measured in the range of 11,000 to 24,000 crawls per referral, against roughly 5 to 1 for traditional search (Cloudflare Radar, 2026).

CrawlerCrawls per referral sent
ClaudeBot~11,000 to 24,000
GPTBot~1,276
PerplexityBot~111
Search~5

Source: derived from Cloudflare Radar, 2026. Network-wide averages; your own ratio will differ.

These are network averages, and your site's ratio is the number that actually matters. A high ratio is not automatically bad. It can mean an assistant is learning your product in depth. But it is a cost worth seeing, and it is the kind of metric you can only compute when you count both sides, the crawl and the referral, in one place. Network data tells you the shape of the problem. First-party data tells you yours.

Did the agent get markdown or HTML?

This is the metric almost no one tracks, and it is the one you can act on fastest. Agents read clean markdown far more reliably than they read browser markup full of layout and scripts. So the useful question is not just "did an agent visit," but "did it get something it could actually use." Caprail grades every response served to an agent by format, markdown or not, per request.

Treat that grade as a new column in your analytics. A page that agents fetch often but always receive as heavy HTML is a page leaving value on the table. The fix is to serve a markdown twin of your content, the same source rendered two ways, so a browser gets the page and an agent gets clean text. Caprail's own pages work exactly this way.

Vendor share is the other moving piece worth watching here, because different agents prefer different formats and crawl at different depths.

CrawlerShare of crawler traffic
Googlebot27%
GPTBot11.5%
Bytespider10.3%
ClaudeBot9.7%

Source: Cloudflare Radar, May 2026. These shares flip month to month; date-stamp them whenever you cite them.

How to measure AI agents on your own site

The short answer is to measure on the server, because that is the only layer every visitor passes through. A browser tool sees a slice. Middleware sees everything: the headless crawler, the live fetcher, the referred human, and the format each one received. That is why Caprail classifies each request in your server before a script could run.

Setup is two lines in your middleware:

import { createCaprailMiddleware } from "@caprail-dev/analytics/next";
 
export const middleware = createCaprailMiddleware();

When we added that to caprail.dev, the first hour surfaced agents the team did not know were visiting, which is the usual experience rather than the exception. From there you get a live feed of agent requests, AI referral attribution, a coverage view of which engines have read which pages, and the readiness audit that flags pages no agent has found yet or that get served in a format agents cannot use. Nothing runs in your visitors' browsers, and nothing slows your pages down.

FAQ

Why doesn't traditional analytics show ChatGPT traffic?

Because it runs in the browser. Script-tag analytics only fires when a real browser executes JavaScript, and most agents never do. As of 2026, Cloudflare Radar reports 57.5 percent of HTML requests come from bots. Referrer-less clicks from assistant apps also land in "direct," hiding the source.

How do I know if AI bots are crawling my website?

Check your server logs or middleware for known agent user agents such as GPTBot, ClaudeBot, and PerplexityBot. A server-side analytics tool does this automatically, classifying each request as crawler, fetcher, browser, search bot, or human. Browser-based tools cannot see these visits at all, since the bots run no script.

What is the difference between an AI crawler and AI referral traffic?

A crawler is a bot that reads your pages directly, often in bulk for training or indexing. AI referral traffic is a human who clicked through to your site from an assistant's answer. One is a machine reading you, the other is a person an assistant sent. Measuring both is the point of agent analytics.

Does AI referral traffic convert better than search?

In published studies it converts much higher per visit. One B2B site saw ChatGPT referrals convert at 15.9 percent against 1.76 percent for search organic (Seer Interactive, 2025), and Ahrefs reported roughly a twenty-fold signup edge (2025). Volume is still small, so treat these as directional rather than guaranteed.

Should I serve markdown or HTML to AI agents?

Serve both. Agents parse clean markdown more reliably than browser markup, so a markdown twin of each page, the same source rendered two ways, gives a browser the full page and an agent clean text. Measuring the format each agent actually received tells you whether your content is reaching them in a usable shape.

Conclusion

AI agents are already part of your traffic, and the analytics you run today was never built to see them. The numbers that matter are who is reading you, what they were served, and which assistants send you people, and all three live on the server. The shift is the same one analytics has always made: you cannot improve what you cannot measure.

Start by making the invisible visible. Add the middleware to your site, watch the live feed, and run the readiness audit. Most sites find an agent they never knew about within the first hour, and the first fixable gap the same day. Caprail is free during the private beta, with no credit card required.

Sources