caprail
A glowing red padlock icon on a dark laptop screen, evoking the access-control question of verifying who is really knocking on your server.

How to verify a crawler is really Googlebot, GPTBot, or Meta

Maximilian Leodolterengineeringagentsdetectionverification

Anyone can put Googlebot in a User-Agent header. The string is self-reported, so a scraper that wants your robots.txt exemptions just claims to be the bot you already trust. It works more often than it should: in DataDome's 2025 measurement, 5.7 percent of traffic claiming to be a known AI crawler was fake, and the ChatGPT-User agent alone was spoofed 16.7 percent of the time (DataDome, 2025). The header tells you what a request says it is. To know whether to believe it, you verify the network identity behind it. This is how we do that in production, vendor by vendor.

Key takeaways

  • The User-Agent is a claim, not proof. In 2025, 5.7 percent of AI-crawler traffic was spoofed, peaking at 16.7 percent for ChatGPT-User (DataDome, 2025).
  • Forward-confirmed reverse DNS (FCrDNS) is the gold standard. Resolve the IP to a hostname, confirm it ends in a vendor domain, then resolve that hostname back to the same IP. An impostor cannot fake it without controlling the vendor's DNS.
  • Vendors disagree on what they publish. Google and Bing support reverse DNS; OpenAI, Anthropic, and Perplexity publish only IP ranges; Meta publishes neither cleanly and forces you to fall back to its network's AS number.
  • Verify conservatively. Only flag a request as spoofed when the IP is provably not the vendor's. Treat every uncertain lookup as unverifiable so you never warn on a slow resolver.

Why isn't the User-Agent enough to trust a crawler?

Because it is a string the client chooses, and not every client is honest. A User-Agent travels as plain text the sender controls, so claiming to be Googlebot costs nothing. The incentive is real: bots that pose as search crawlers get waved past rate limits, paywalls, and robots.txt rules written to welcome indexing. DataDome reports seeing more than a million hits per day from fake Googlebots across the sites it protects (DataDome, 2025).

It is not only anonymous scrapers. In August 2025, Cloudflare reported that Perplexity ran undeclared "stealth" crawlers that masqueraded as a generic Chrome browser to fetch pages from sites that had blocked its official bots, an estimated 3 to 6 million such requests a day across tens of thousands of domains. Cloudflare removed Perplexity from its verified-bot program over it (Cloudflare, 2025). Perplexity disputed the finding. The lesson holds either way: if a well-funded vendor's user agent can be in question, an arbitrary one deserves no trust at all.

So the header is fine for measurement, where a wrong guess costs you a slightly off chart. It is not fine for enforcement, where a wrong guess means you blocked a real customer or let a scraper drain your origin. For the substring-matching layer this sits on top of, see how to detect AI agents server-side.

What actually proves a crawler's identity?

The network it comes from, which a spoofer does not control. Two signals carry the proof. The stronger one is forward-confirmed reverse DNS, or FCrDNS: take the request's IP, look up its reverse-DNS (PTR) hostname, confirm that hostname ends in a domain the vendor owns, then do a forward lookup of that hostname and confirm it resolves back to the same IP. A scraper can set any PTR record on its own IP, but it cannot make that PTR forward-resolve back from inside the vendor's DNS. The round trip is the part that cannot be faked.

A dark server room with rows of networking equipment and status lights, where each crawler request resolves to a real IP that either belongs to the vendor or does not.

The second signal is the published IP range. Most major vendors release the address blocks their crawlers operate from, often as a machine-readable JSON file. If the request IP falls inside a current published range, the claim checks out. Ranges are a weaker guarantee than FCrDNS, because they are only as fresh as the list and can lag a vendor's infrastructure changes, but they cover the vendors that offer no reverse-DNS scheme at all. In our pipeline reverse DNS is authoritative and the range list is the fallback, in that order.

Forward-confirmed reverse DNS is the same mechanism mail servers have used for decades to fight sender forgery. It works here for the identical reason: control of a domain's authoritative DNS is exactly the thing an impostor lacks.

How do you verify Googlebot?

You confirm the IP reverse-resolves into a Google domain, then forward-confirm it. Google documents the exact method: a genuine crawler IP has a PTR record ending in googlebot.com, google.com, or googleusercontent.com, and a forward lookup of that hostname returns the original IP (Google, 2026). Here is the forward-confirm in TypeScript, close to what we run:

import { Resolver } from "node:dns/promises";
 
const GOOGLE_SUFFIXES = [".googlebot.com", ".google.com"];
 
async function isForwardConfirmed(ip: string, suffixes: string[]) {
  const resolver = new Resolver({ timeout: 3_000, tries: 1 });
 
  // 1. Reverse lookup: what hostname does this IP claim?
  const hostnames = await resolver.reverse(ip);
 
  // 2. Keep only hostnames under a vendor-owned domain.
  const candidates = hostnames.filter((h) =>
    suffixes.some((s) => h.toLowerCase().endsWith(s)),
  );
  if (candidates.length === 0) return false; // a PTR exists, but not Google's
 
  // 3. Forward lookup must resolve back to the same IP.
  for (const host of candidates) {
    const forward = await resolver.resolve4(host);
    if (forward.includes(ip)) return true;
  }
  return false; // suffix matched but never forward-confirmed: a spoofed PTR
}

If reverse DNS is unavailable, fall back to Google's published ranges. One sharp edge to flag: as of March 2026 Google renamed that file from googlebot.json to common-crawlers.json and moved it to https://developers.google.com/crawling/ipranges/common-crawlers.json (Google Search Central, 2026). If you copied an older snippet, update the path or your range check silently breaks. The reverse-DNS domains did not change.

How do the other vendors let you verify?

Inconsistently, which is the real headache. Each vendor picks its own mechanism, so a verifier has to carry a different strategy per vendor rather than one universal check. Google and Bing offer reverse DNS plus ranges. OpenAI, Anthropic, and Perplexity publish IP ranges but no reverse-DNS scheme. Meta publishes no clean crawler IP list at all. Here is the map we built our checks against:

VendorReverse DNSPublished IP rangesNotes
GoogleYescommon-crawlers.json (+ special crawlers)Strongest: both signals, well documented
MicrosoftYesbingbot.jsonrDNS to search.msn.com, plus a range list
OpenAINogptbot.json, searchbot.json, moreRanges only; the IPs live in Azure space
AnthropicNoTwo ranges in docs, no JSON endpointSmallest footprint, no machine-readable file
PerplexityNoperplexitybot.json, perplexity-user.jsonRanges only, after the 2025 stealth episode
MetaPartialNone publishedDerive from network AS number; messiest case

Meta is the one that breaks the pattern. It publishes no crawler IP file, so the most reliable signal is the autonomous system the request originates from: Meta's network is AS32934, and traffic outside it is not Meta. Some Meta infrastructure IPs also reverse-resolve into Meta-owned domains, but there is no documented FCrDNS scheme for meta-externalagent the way Google documents one for Googlebot. When a vendor gives you the least, you verify with the coarsest tool that still holds, and you are honest in the UI about how strong that signal is. For why the bot's purpose, not just its name, should drive what you do next, see crawler vs fetcher vs AI browser.

How does Caprail verify crawlers in production?

With a deliberately cautious contract: every check returns verified, mismatch, or unverifiable, and only a provable mismatch ever warns. This is the rule we settled on after watching early versions cry wolf. A slow resolver or a vendor with no published signal must never produce a false spoof alert, because one bad warning teaches you to ignore all of them. So the logic only returns mismatch when a published range exists and the IP sits outside it, or when DNS positively answered with a non-vendor hostname. Any timeout, transport error, or unconfigured vendor returns unverifiable, which is silent.

// Reverse DNS is authoritative when it confirms; ranges are the fallback.
if (config.rdnsSuffixes.length > 0) {
  const rdns = await rdnsCheck(addr, config.rdnsSuffixes);
  if (rdns === "confirmed") return "verified";
}
if (config.cidrs.length > 0) {
  if (ipInAnyCidr(addr, config.cidrs)) return "verified";
  return "mismatch"; // range list is authoritative and the IP is outside it
}
// rDNS-only vendor: a mismatch needs DNS to have actually answered, not timed out.
return rdns === "denied" ? "mismatch" : "unverifiable";

The check never runs on the request path. When an agent request lands, ingest seeds it as pending with a single in-memory set lookup, no network call, so the response is never delayed by a DNS round trip. A background cron then drains the queue, deduplicating by IP and vendor so each address is resolved once and the result fans out to every row that shares it. When a mismatch lands, the dashboard shows a small warning marker next to the bot, reading "Source IP does not match this vendor. Possible spoofed bot." That is the whole point of separating measurement from enforcement, made visible. The verified identity then feeds the rest of the picture, covered in analytics for AI agents.

When should you actually run verification?

When the decision has teeth, not on every hit. Verification adds a DNS round trip, and doing it inline on every request would put a slow external lookup in front of your page. Reserve the cost for choices where a wrong call is expensive: blocking, rate-limiting, granting a paywall exemption, or flagging traffic as fraudulent. For plain measurement, the substring match on the header is enough, because a miscounted bot in a chart is cheap.

The economics make the split obvious. AI crawling is no longer a rounding error: across Cloudflare's network in 2025, Googlebot generated about 4.5 percent of HTML requests and all AI bots combined averaged about 4.2 percent, while live "user-action" fetching grew more than fifteenfold over the year (Cloudflare, 2025). As that share climbs, so does the payoff for impersonating it, which is exactly why the gap between a claimed identity and a verified one keeps widening. We broke down what that traffic looked like across our own beta in the Q2 crawler share report.

All AI crawlers

5.7%

ChatGPT-User

16.7%

Source: DataDome 2025 Global Bot Security Report. Share of traffic claiming each identity that was found to be spoofed.

For the broader question of whether to let these crawlers through once you can tell them apart, see should you block AI crawlers.

FAQ

How do I verify a request is really Googlebot?

Run a reverse-DNS lookup on the request IP and confirm the hostname ends in googlebot.com, google.com, or googleusercontent.com, then do a forward lookup of that hostname and confirm it returns the same IP (Google, 2026). If reverse DNS is missing, check the IP against Google's published ranges, now served as common-crawlers.json.

Can a User-Agent header be faked?

Completely. It is a self-reported string the client sets to anything it wants, which is why DataDome found 5.7 percent of AI-crawler traffic was spoofed in 2025, rising to 16.7 percent for ChatGPT-User (DataDome, 2025). Treat the header as a claim, and verify the IP before acting on anything that matters.

What is forward-confirmed reverse DNS?

It is a two-step identity check. You resolve an IP to its reverse-DNS hostname, confirm that hostname belongs to the vendor's domain, then resolve the hostname forward and confirm it points back to the original IP. The round trip is what defeats spoofing: a scraper can set its own PTR record but cannot forge the forward resolution inside the vendor's DNS.

How do you verify OpenAI's GPTBot if there is no reverse DNS?

You check the source IP against OpenAI's published ranges. OpenAI lists separate JSON files for GPTBot, OAI-SearchBot, and ChatGPT-User, and a genuine request falls inside the matching block. There is no reverse-DNS option, so the range list is your only signal, which means keeping that snapshot current matters more for OpenAI than for Google.

Does verifying every crawler slow down my site?

It does if you do it inline, because each check adds a DNS lookup. The fix is to verify off the request path: seed each request for verification with a cheap in-memory check, then resolve identity in a background job. We deduplicate by IP and vendor so each address is looked up once, and the page response never waits on DNS.

Conclusion

A User-Agent is a name tag a visitor writes for themselves. For counting traffic, that is fine. For any decision with consequences, you need the part the visitor cannot forge: a network identity that forward-confirms back to the vendor, or an IP inside a range that vendor published. Verify reverse DNS where it exists, fall back to ranges where it does not, and treat Meta and the range-only vendors as the weaker cases they are.

Then stay conservative. Only call a request spoofed when it is provably not who it claims, keep the check off the hot path, and let uncertainty stay quiet. That is the contract we ship in Caprail: every crawler classified server-side, its identity verified in the background, and a clear warning the moment a source IP does not match the vendor it claims. It is free during the private beta, with no credit card required. See who is really reading you.

Sources