caprail
A close-up of a barbed wire fence against an open sky, evoking the decision of whether to gate AI crawlers out of your site.

Should you block AI crawlers in 2026? A traffic-data decision guide

Maximilian Leodoltercrawlersagentsdata

Bots now make up more than half of all web traffic, the first time machines have outnumbered people in a decade (Imperva, 2025). A growing slice of that is AI crawlers, and Cloudflare now blocks them by default for new domains (Cloudflare, 2025). So the question lands on every site owner's desk: should you block AI crawlers too? This guide gives you a decision rule, the traffic data behind it, and the one thing you need before you can choose at all.

Key takeaways

  • Do not block all AI bots. The right unit of decision is crawler purpose, not the label "AI."
  • Training crawlers take a lot and send nothing back. Anthropic's crawler hit roughly 70,000 reads per referral in 2025 (Cloudflare, 2025).
  • Retrieval fetchers send people who convert. ChatGPT-referred visits converted at 15.9 percent against 1.76 percent for search (Seer Interactive, 2025).
  • You cannot make this call blind. You first need server-side visibility into which bots actually hit you and why.

Should you block AI crawlers?

Not as one undifferentiated group. The useful answer in 2026 is to decide by what a crawler does, because AI bots split into two kinds with opposite economics. Some read your pages to train a model and send no traffic back. Others fetch a page live to answer a question and pass a referral to you. Cloudflare's network data shows model-training crawling peaked at seven to eight times the volume of search crawling in 2025 (Cloudflare, 2025), so the two are not close to equal in scale.

The one-line rule: block or charge the crawlers that only take, allow the ones that send people, and treat search bots as the baseline you almost never touch. That sounds simple, and the logic is. The hard part is knowing which bucket each request falls into, which is where most sites are flying blind. For the underlying bot taxonomy, see crawler vs fetcher vs AI browser.

What do AI crawlers actually cost you?

Real money, mostly in bandwidth and origin load. Wikimedia reported a 50 percent rise in multimedia download bandwidth since January 2024, driven largely by AI scrapers, with bots accounting for 65 percent of its most expensive traffic while making up only about 35 percent of pageviews (Wikimedia Foundation, 2025). The cost is not spread evenly. It clusters on the heavy, uncached requests that bots favor.

A humanoid robot standing against a plain background, representing the automated AI crawlers that read a site's pages.

Scale this down to a normal site and the pattern holds. Bots are 51 percent of all traffic and bad bots alone are 37 percent (Imperva, 2025). On Cloudflare's network, AI bots averaged 4.2 percent of HTML requests across 2025, with Googlebot adding another 4.5 percent on its own (Cloudflare, 2025). None of that is catastrophic by itself. It is a tax that grows quietly, and one you should be able to read on a bill.

Humans 49%

14%

Bad bots 37%

Humans

Good bots

Bad bots

Source: Imperva 2025 Bad Bot Report. Share of all web traffic, 2024. Good bots are the remainder of total bot traffic after bad bots.

The upside: crawlers that send people back

The reason not to wall everything off is that some AI traffic converts unusually well. One B2B study recorded ChatGPT-referred visits converting at 15.9 percent and Perplexity at 10.5 percent, against 1.76 percent for search organic (Seer Interactive, 2025). Those visitors also looked deeper, at 2.3 pages per session versus 1.2 for organic. They arrive pre-qualified, because an assistant already recommended you before they clicked.

ChatGPT

15.9%

Perplexity

10.5%

Search

1.76%

Source: Seer Interactive, 2025, single B2B site. Conversion rate by traffic source. Treat as directional.

The catch is volume. This is high-value traffic, not high-traffic traffic. Block the assistants that send these visits and you do not save much bandwidth, but you do shut off your best-converting channel. That asymmetry is exactly why a blanket block is the wrong move. For the mechanics of capturing this traffic, see how to track AI referral traffic.

The crawl-to-referral ratio: the number that decides

If you read only one metric before choosing, read this one. The crawl-to-referral ratio is how many times a crawler reads your pages for each visitor it sends back, and in 2025 it varied by four orders of magnitude across providers (Cloudflare, 2025). Google Search sat near 3:1 rising toward 30:1. Anthropic's crawler peaked around 500,000 reads per referral and settled into the tens of thousands. Same web, wildly different bargains.

Google

~30:1

Perplexity

~700:1

OpenAI

~3,700:1

Anthropic

~70,000:1

log scale, reads per referral

Source: Cloudflare Radar 2025 Year in Review. Approximate 2025 crawl-to-referral ratios, log scale (each gridline is 10x). Higher means more reads per visitor sent back.

A low ratio is a fair trade: the bot reads you and sends people in roughly equal measure. A ratio in the tens of thousands is not a trade at all. It is extraction. The ratio is the cleanest signal for where to spend a block, a charge, or an allow, and it can shift quarter to quarter, so it is worth watching rather than setting once.

Training crawlers and live fetchers are not the same bot

Treating them alike is the most common mistake, and it is costly in both directions. Training crawls and live fetches behave differently and serve different ends. Cloudflare measured user-action fetching, the live kind that powers an answer, growing roughly 21x across 2025, while training crawling stayed the dominant share of volume (Cloudflare, 2025). One feeds a model you may never benefit from. The other is a customer mid-question.

The insight most block lists miss: "AI bot" is not a useful category for a blocking decision. A training crawler and a retrieval fetcher can carry similar user agents and come from the same company, yet one is pure cost and the other is pre-sales. Block by user agent and you will inevitably catch the fetcher you wanted to keep.

If you block by name alone, you will eventually block the fetcher that was about to send a buyer, while leaving a training crawler untouched because its token looked friendlier. The distinction has to be made on purpose, not on the label. We break this down further in crawler vs fetcher vs AI browser.

You can't decide what you can't see

Here is the part most advice skips. Every rule above assumes you know which crawlers hit you, how often, and to what end, and most sites do not. Worse, the data cannot be trusted at face value: Cloudflare caught Perplexity using an undeclared, browser-impersonating crawler that pulled content from test domains which disallowed all crawling in robots.txt (Cloudflare, 2025). A robots.txt line is a request, not a wall.

Rows of illuminated server racks in a data center, the infrastructure that absorbs the bandwidth cost of AI crawling.

So a robots.txt entry tells you what you asked for, not what happened. The only reliable record is server-side: classify every request from the raw signals as it arrives, before any script runs or any directive is politely ignored. That is the gap Caprail fills, naming each crawler and separating the training reads from the live fetches from the human referrals. For the broader case, see analytics for AI agents.

What our own beta data showed

In our first two beta weeks across two sites, crawler reads outnumbered referred human visits by roughly ten to one: about 151 classified crawler events against 15 assistant-referred people. That lopsidedness, on a tiny sample, is the network-wide story in miniature: the machines read far more than they send. The point is not the exact number. It is that the split was visible at all, named bot by bot, which is the precondition for deciding anything. The fuller snapshot, crawlers and referrals together, is in our Q2 crawler share report.

A decision framework you can act on

Pull it together and the choice stops being all-or-nothing. Cloudflare's pay-per-crawl, launched July 2025, even lets you answer some crawlers with an HTTP 402 and a price instead of a flat allow or block (Cloudflare, 2025). Match the response to the crawler's purpose and ratio:

Crawler typeSends referrals?Default moveWhy
Search bots (Googlebot)Yes, low ratioAllowThe trade is fair and the traffic is large
Retrieval fetchers (live answers)YesAllowPre-qualified visitors that convert well
Training crawlers (high ratio)NoBlock or chargePure cost unless a deal or citation value justifies it
Undeclared or stealth crawlersNoBlockThey ignore your rules already

Two caveats keep this honest. First, being read by a training crawler can still buy you visibility inside an assistant's answer later, so a high ratio is a reason to charge or watch, not always to block outright. Second, none of these rows mean anything until you can see which bucket your real traffic falls into. Decide on evidence from your own logs, not on a generic list.

FAQ

Should I block all AI crawlers in robots.txt?

No. Blocking all AI bots also blocks the retrieval fetchers that send pre-qualified visitors, who converted at 15.9 percent in one study against 1.76 percent for search (Seer Interactive, 2025). Block by purpose instead: charge or block training-only crawlers, and keep the fetchers that bring people.

Does blocking AI crawlers hurt my visibility in ChatGPT or Perplexity?

It can. If you block the live fetchers that read a page to answer a question, you can drop out of those answers, and the referral traffic with it. Training crawlers are different: blocking them mainly protects bandwidth. The two need separate decisions, which is why classifying crawler purpose matters more than a single on-off switch.

How much traffic do AI crawlers actually consume?

On Cloudflare's network, AI bots averaged 4.2 percent of HTML requests across 2025 (Cloudflare, 2025). The load is uneven, though: Wikimedia found bots drove 65 percent of its most expensive traffic while being about 35 percent of pageviews (Wikimedia Foundation, 2025). The cost concentrates on heavy, uncached requests.

Do AI crawlers respect robots.txt?

Not always. Cloudflare documented Perplexity using an undeclared crawler that impersonated a browser to pull content from domains that disallowed all crawling (Cloudflare, 2025). Treat robots.txt as a polite request, and rely on server-side measurement to see what actually reaches your origin.

What is a crawl-to-referral ratio and why does it matter?

It is the number of times a crawler reads your pages for each visitor it sends back. In 2025 it ranged from about 3:1 for Google Search to tens of thousands to one for Anthropic's crawler (Cloudflare, 2025). A low ratio is a fair exchange. A very high one is extraction, and the clearest signal of where a block or a charge belongs.

Conclusion

The honest answer to "should you block AI crawlers" is: block the ones that take and give nothing, keep the ones that send you people, and stop treating "AI bot" as a single thing. The economics are not subtle. Some crawlers send a buyer for every visit, and some take 70,000 reads for every visitor they return.

But the framework only works if you can see your own traffic, named and split by purpose, rather than guessing from a robots.txt file that the busiest crawlers may ignore anyway. Start there. Caprail classifies every crawler, fetcher, and referral server-side, and it is free during the private beta with no credit card required. See what is reading you, then decide.

Sources