Council of AI — agent infrastructure
What changed when Cloudflare started blocking AI agents
In 2026, Cloudflare introduced tools that let site owners control AI traffic by category — Search, Agent, and Training. Mixed-use crawlers that blend all three behind a single user agent are now blocked by default on ad-supported sites unless they meet transparency requirements. This page explains what happened and why signed agent identity matters.
What changed
July 1, 2026 — AI bot policy categories
Cloudflare launched three separate controls for AI traffic: Search (crawlers that index content for search engines), Agent (crawlers that act on behalf of users), and Training (crawlers that collect content for model training). Site owners can allow, block, or disallow each category independently.
Source: Cloudflare blog, Aug 21, 2026
August 21, 2026 — Bot Preference Sync
A new feature that automatically syncs a site's robots.txt with the AI bot policies configured in the Cloudflare dashboard. For new customers, Bot Preference Sync is on by default. For ad-supported sites, Training defaults to Disallow.
The transparency requirement
Mixed-use crawlers that blend Search, Agent, and Training behind a single user agent must provide transparency to avoid being blocked. Specifically, they must:
- Respect a "no training" preference in robots.txt
- Give site owners a way to opt out of AI summaries
- Provide URL-level visibility into which pages were used for training vs search
- Show publicly that disallowing training does not hurt traditional search results
Crawlers that don't meet these requirements are blocked by default when a site owner sets "Disallow Training." Transparency is the price of admission.
What it means for agents
Agent traffic is being reclassified at the network layer. A legitimate agent that fetches data on behalf of a user looks identical to a training crawler unless it can prove its identity and purpose. The three categories (Search, Agent, Training) create a new classification layer, and agents that can't demonstrate which category they belong to get blocked with the training crawlers.
This is not a hypothetical. Cloudflare's AI Bot Transparency tracker on Radar publicly shows which bots meet the transparency requirements and which don't.
How signed identity helps
An agent that carries a signed, verifiable identity card — whether through A2A agent cards, ERC-8004 on-chain registration, or SCITT-anchored logs — can distinguish itself from anonymous crawlers. The identity card says who the agent is, what it does, and who signed the claim. A site owner or network layer can verify the signature without asking the agent's operator.
This is the same principle as the GSPC measurement board: a signed artifact that anyone can verify without asking us. The difference is that the artifact describes the agent itself, not a measurement result.
The numbers
agents registered on ERC-8004 across 24 chains (Sep 11, 2026). New-cohort median trust: "Elevated Risk" (Chainaware).
Source: Chainaware, Sep 11, 2026
Search, Agent, Training — the new classification layer. Agents that can't demonstrate their category get blocked with training crawlers.
Source: Cloudflare, Jul 1, 2026
Our position
We measure agent behaviour, we don't certify agents. An agent that carries a signed measurement card has demonstrated something about its behaviour on a frozen instrument. That is evidence, not a certificate. Cloudflare's blocking change makes signed identity more valuable — not because we sell it, but because it's the mechanism that lets legitimate measured agents distinguish themselves from anonymous crawlers.
This page is a CSOAI editorial act. Cloudflare does not endorse this page. Sources: Cloudflare blog (blog.cloudflare.com), Cloudflare Radar (radar.cloudflare.com).