caprail
A dark server rack with networking gear and an orange patch cable in a data center, where every AI agent request first arrives.

How to detect AI agents server-side from HTTP headers

Maximilian Leodolterengineeringagentsdetection

Most sites cannot see the agents reading them. AI bots already account for 4.2 percent of HTML requests on Cloudflare's network, on top of Googlebot's own 4.5 percent (Cloudflare, 2025), and that share keeps climbing. The catch: nearly all of it is invisible to the analytics most teams run, because that runs in the browser and agents do not open one. The fix is to classify each request where it actually lands, on your server, from the headers it arrives with. This guide shows you how, with the same logic we ship in production.

Key takeaways

  • Client-side analytics misses agents by design. Bots do not run your JavaScript, so detection has to happen server-side, in middleware or a request handler.
  • The User-Agent header is the primary signal. Match known tokens as case-insensitive substrings, most-specific-first, and bucket each into one of five types.
  • User agents are spoofable. Cloudflare caught Perplexity using undeclared crawlers that impersonated Chrome (Cloudflare, 2025), so high-stakes calls need IP or reverse-DNS verification on top.
  • You can ship a working classifier in about 40 lines, or two lines of middleware if you would rather not maintain the token list yourself.

Why detect AI agents server-side at all?

Because the alternative cannot see them. Bots are now most of the web: automated traffic hit 51 percent of all requests in 2024, with bad bots alone at 37 percent (Imperva, 2025). Almost none of that registers in a tool like GA4, which fires from a browser script. A classic crawler requests your HTML and leaves. It never executes the JavaScript that would log the visit, so the agent reads your page and your analytics records nothing.

Server-side detection closes that gap. Every request, human or agent, passes through your server before anything renders, so that is the one place you can classify all of it from the raw headers. The payoff is concrete: you learn which models train on you, which assistants fetch you live to answer a question, and which ones send you nothing back. For the wider argument, see why traditional analytics can't see AI agents.

A gigabit network switch with yellow and red Ethernet cables plugged into active ports, representing the raw requests that reach your server before any page renders.

What signal actually identifies an AI agent?

The User-Agent request header, first and foremost. Almost every well-behaved AI bot announces itself with a stable token: GPTBot for OpenAI's training crawler, ClaudeBot for Anthropic's, PerplexityBot, Google-Extended, meta-externalagent, and so on. These are published and reasonably stable, which makes a substring match the workhorse of detection. In the Cloudflare unique-pages data for late 2025, Googlebot led at 11.6 percent, with GPTBot at 3.6 percent and ClaudeBot at 2.4 percent (Cloudflare, 2025).

Googlebot

11.6%

GPTBot

3.6%

Bingbot

2.6%

Meta-ExternalAgent

2.4%

ClaudeBot

2.4%

Source: Cloudflare 2025 Radar Year in Review. Share of unique web pages crawled, Oct to Nov 2025.

According to Cloudflare's 2025 network data, AI crawlers reached 4.2 percent of HTML requests while the broader AI-crawler category made up roughly 20 percent of all verified-bot traffic (Cloudflare, 2025). The header is the cheapest, highest-coverage signal you have. It is not the only one, and it is not tamper-proof, but it is where every classifier should start.

How do you write the classifier?

You walk a list of known tokens, most-specific-first, and return the first case-insensitive substring hit. That ordering is the part people get wrong. Here is a compact TypeScript version of the exact approach we run in production:

type AgentType = "crawler" | "fetcher" | "browser" | "search" | "human";
 
// Declared MOST-SPECIFIC-FIRST: qualified tokens must precede shorter relatives,
// so "Claude-User" matches before "ClaudeBot", "ChatGPT-User" before "GPTBot".
const AGENTS: {
  token: string;
  name: string;
  vendor: string;
  type: AgentType;
}[] = [
  {
    token: "Claude-User",
    name: "Claude-User",
    vendor: "Anthropic",
    type: "fetcher",
  },
  {
    token: "ClaudeBot",
    name: "ClaudeBot",
    vendor: "Anthropic",
    type: "crawler",
  },
  {
    token: "ChatGPT-User",
    name: "ChatGPT-User",
    vendor: "OpenAI",
    type: "fetcher",
  },
  {
    token: "OAI-SearchBot",
    name: "OAI-SearchBot",
    vendor: "OpenAI",
    type: "search",
  },
  { token: "GPTBot", name: "GPTBot", vendor: "OpenAI", type: "crawler" },
  {
    token: "Google-Extended",
    name: "Google-Extended",
    vendor: "Google",
    type: "crawler",
  },
  { token: "Googlebot", name: "Googlebot", vendor: "Google", type: "search" },
  {
    token: "Perplexity-User",
    name: "Perplexity-User",
    vendor: "Perplexity",
    type: "fetcher",
  },
  {
    token: "PerplexityBot",
    name: "PerplexityBot",
    vendor: "Perplexity",
    type: "search",
  },
  {
    token: "meta-externalfetcher",
    name: "meta-externalfetcher",
    vendor: "Meta",
    type: "fetcher",
  },
  {
    token: "meta-externalagent",
    name: "meta-externalagent",
    vendor: "Meta",
    type: "crawler",
  },
];
 
export function classify(ua: string | null | undefined) {
  if (!ua)
    return { name: null, vendor: null, type: "human" as const, isAgent: false };
  const haystack = ua.toLowerCase();
  for (const agent of AGENTS) {
    if (haystack.includes(agent.token.toLowerCase())) {
      return {
        name: agent.name,
        vendor: agent.vendor,
        type: agent.type,
        isAgent: true,
      };
    }
  }
  return { name: null, vendor: null, type: "human" as const, isAgent: false };
}

The ordering rule is not cosmetic. Claude-User (a live fetcher answering someone's question) contains no overlap with ClaudeBot, but Google-Extended and GoogleOther both contain Google, and Gemini's URL fetcher sends a User-Agent that is literally just Google. Match in the wrong order and you label a live, buyer-adjacent fetch as a training crawl, or you swallow every Google service under one bucket. Declare the qualified tokens first, keep the bare ones last, and reserve exact-match for the ambiguous singletons like Google.

Why isn't the user agent enough on its own?

Because anyone can set it to anything. The User-Agent is a self-reported string, and not every agent is honest. Cloudflare documented Perplexity running undeclared crawlers that impersonated a Chrome-on-macOS browser, rotated through unlisted IP ranges, and pulled content from domains that disallowed all crawling in robots.txt (Cloudflare, 2025). Cloudflare delisted them from its verified-bot program over it. So what do you do when the header lies?

You verify identity for the cases that matter. The major honest crawlers publish how: confirm Googlebot or GPTBot with a reverse DNS lookup on the request IP, then a forward lookup back to confirm it resolves to the published domain. Vendors that publish IP ranges let you check membership directly. You do not need this for every request. Reserve it for decisions with teeth, like blocking or rate-limiting, and let the cheap substring match handle everyday measurement.

The trap most block lists fall into: treating the user agent as proof rather than a claim. A spoofed string sails straight through a substring check. The header tells you what a request says it is, and IP verification tells you whether to believe it. Measurement can trust the claim; enforcement should not.

For why purpose matters more than the label, see crawler vs fetcher vs AI browser.

What are the five agent types, and why bucket them?

Because "AI bot" is too coarse to act on. The same vendor sends bots with opposite economics, so a useful classifier sorts every request into one of five buckets. One reads you to train a model and sends nothing back. Another fetches you live to answer a question, and that visitor converts. ChatGPT-referred visits converted at 15.9 percent in one B2B study, against 1.76 percent for search organic (Seer Interactive, 2025). Lumping the training crawler in with the fetcher hides exactly the distinction you care about.

TypeWhat it doesExamplesSends traffic back?
crawlerReads pages to train or index a modelGPTBot, ClaudeBot, Google-ExtendedNo
fetcherFetches a page live to answer one queryChatGPT-User, Claude-User, Perplexity-UserOften, as a referral
searchIndexes for a search or answer engineGooglebot, PerplexityBot, OAI-SearchBotYes, via results
browserAgentic browser acting for a userChatGPT AgentSometimes
humanNo known agent tokenRegular visitorsn/a

With the type attached, your dashboard stops saying "1,000 bot hits" and starts saying "850 training reads, 120 live fetches, 30 search crawls." That second sentence is one you can make a decision from. It is what powers the block-or-allow decision guide.

Where do you run detection in a real app?

In middleware or your request handler, before the response is built. In Next.js that means a single middleware file. The naive version is just the classifier above wired to the incoming request:

// middleware.ts
import { NextResponse, type NextRequest } from "next/server";
import { classify } from "./lib/classify";
 
export function middleware(req: NextRequest) {
  const result = classify(req.headers.get("user-agent"));
  if (result.isAgent) {
    // log it, tag it, rate-limit it, whatever you need
    console.log(
      `${result.vendor} ${result.name} (${result.type}) → ${req.nextUrl.pathname}`,
    );
  }
  return NextResponse.next();
}
 
export const config = {
  matcher: ["/((?!_next/static|_next/image|favicon.ico).*)"],
};

One sharp edge we hit building this: middleware runs before the response, so it cannot read the final Content-Type. That matters because a big question for the agentic web is whether bots got HTML or your markdown twin, and you only know the served format after the handler runs. On the edge you fall back to inferring format from the path. A platform that wraps the actual response, like a Cloudflare Worker, sees the real Content-Type directly. It is a small thing that quietly changes which signals you can collect where.

Syntax-highlighted source code on a dark screen, the detection logic that classifies each request as it arrives.

How do you keep the token list current without owning it?

You let something else maintain it, because the list moves constantly. New tokens appear, vendors rename them, and speculative ones like xAI's Grok need watching before they go live. Keeping a hand-written list accurate is a standing chore. This is the part we turned into the product: @caprail-dev/analytics ships the classifier, the maintained token list, and the five-type taxonomy, and beacons each request without blocking the response.

// middleware.ts: the whole integration
import { createCaprailMiddleware } from "@caprail-dev/analytics/next";
 
export const middleware = createCaprailMiddleware();
 
export const config = {
  matcher: ["/((?!_next/static|_next/image|favicon.ico).*)"],
};

According to our own measurement, that two-line collector classifies every request server-side, names the bot and vendor, and tags the type, the same pipeline behind the charts above. The middleware beacons via the runtime's keep-alive hook, so a slow or failing collector never touches your response. For the full picture of what to do with the data, see analytics for AI agents.

What we saw in our own logs

In our first two beta weeks across two sites, server-side classification logged crawler reads outnumbering referred human visits by roughly ten to one: about 151 classified crawler events against 15 assistant-referred people. On a tiny sample, that is the network-wide pattern in miniature, the machines read far more than they send. The exact number is not the point. The point is that the split was legible at all, named bot by bot, which no client-side tool could have shown us. The fuller snapshot is in our Q2 crawler share report.

FAQ

Can you detect AI agents from the User-Agent header alone?

Mostly, yes, for measurement. Most honest AI bots send a stable token like GPTBot or ClaudeBot, so a substring match catches them. But user agents are spoofable, and Cloudflare found Perplexity using crawlers that impersonated Chrome (Cloudflare, 2025). For blocking decisions, add IP or reverse-DNS verification.

Why can't Google Analytics track AI bots?

Because it runs in the browser. Tools like GA4 fire a JavaScript tag, and crawlers request your HTML without executing any script. The agent reads the page and the tag never runs, so the visit is never recorded. Bots were 51 percent of web traffic in 2024 (Imperva, 2025), almost all invisible to client-side analytics. Detection has to happen server-side.

What is the difference between a crawler and a fetcher?

A crawler reads pages in bulk to train or index a model and usually sends no traffic back, like GPTBot. A fetcher pulls one page live to answer a user's question and often passes a referral, like ChatGPT-User. The distinction is commercial: fetcher-referred visits converted at 15.9 percent versus 1.76 percent for search in one study (Seer Interactive, 2025).

How do you verify a crawler is really Googlebot?

Run a reverse DNS lookup on the request IP, confirm it resolves to the vendor's published domain such as googlebot.com, then do a forward lookup to confirm that hostname maps back to the same IP. Vendors that publish IP ranges let you check membership directly. Do this for enforcement decisions, not for every request, since it adds a lookup to the path.

Does server-side detection slow down my site?

It should not. Classifying a user agent is a string scan over a short list, which is microseconds. The cost to avoid is the network beacon: never block the response on it. Caprail's middleware sends each event through the runtime's keep-alive hook, so a slow or failing collector never delays the page the visitor or agent is waiting for.

Conclusion

Detecting AI agents is not exotic. It is a substring match on a header you already receive, sorted into types that tell you what each bot is actually doing. Start there: classify the User-Agent server-side, bucket every request into crawler, fetcher, search, browser, or human, and you can already see a layer of traffic that client-side analytics was never able to show you.

Then harden the parts that matter. Verify identity by IP where the decision has teeth, and let something maintain the token list so you are not chasing renames every month. That last part is what we built Caprail to do. It classifies every crawler, fetcher, and referral server-side, in two lines of middleware, and it is free during the private beta with no credit card required. See what is reading you.

Sources