Cloudflare Blocks AI Agents: A Tested Fallback (2026)

Cloudflare now treats AI agents as a separate, blockable category from search and training crawlers. Since September 15, 2026, new ad-monetized sites default to blocking that Agent behavior on pages with ads, and existing sites that blocked AI bots before get migrated to the same default automatically¹.

If your agent has a tool that fetches web pages, that's the change that affects your code, not the "mixed-use crawler" story most coverage led with.

TL;DR

Cloudflare's Bot Management now recognizes "Agent" as its own bot behavior, distinct from Search and Training: "user-directed agents visiting a page on behalf of a human, such as chat fetch bots and browser-use agents"¹². As of September 15, 2026, new ad-monetized domains default to blocking that behavior on pages that serve ads, and existing domains that had the old "Block AI" setting on get migrated the same way¹. This post builds and runs a small Python web-fetch tool that identifies itself honestly, checks robots.txt before fetching, and treats a 403 and a 402 response as two different, meaningful outcomes — instead of retrying with a different User-Agent, which is exactly the behavior this policy exists to push back on.

What you'll learn

• What changed for AI agents specifically on September 15, 2026, and how it's different from the "block AI crawlers" story most coverage led with
• Why "Agent" is its own bot behavior, separate from the "mixed-use crawler" (Search + Training) problem the same announcement spent most of its words on
• Which real, Verified bots fall under Agent behavior today, by name and operator
• A tested, dependency-free Python pattern for a web-fetch tool that checks robots.txt, identifies honestly, and handles 403 vs. 402 as distinct outcomes
• Why spoofing a Verified bot's User-Agent to dodge a block is a worse idea now than it was a year ago
• What this means practically if you're building your own agent, not shipping one at OpenAI or Anthropic's scale

Three behaviors, not one "block AI" switch

Most coverage of Cloudflare's September 15 announcement led with "mixed-use crawlers" — a single crawler, like Googlebot, that does both search indexing and AI training, which historically forced site owners into an all-or-nothing choice. Cloudflare's fix for that is a new Disallow AI Training setting, and a new Accountable designation. Apple, Google, and Microsoft earn it for their mixed-use crawlers by providing opt-outs, transparency, and a guarantee that disallowing training won't hurt search rankings; Amazon, Anthropic, Meta, and OpenAI earn the same designation for their separate, training-only crawlers, since keeping search and training on two different bots means blocking one was never going to touch the other¹.

That's real, and it's the headline. But it's not the part that matters if you're the one building the agent, rather than the one publishing the pages an agent might fetch.

Since July 1, 2026, Cloudflare has classified bot traffic into three independently controllable behaviors¹³²:

Search and Training get most of the press because that's where the "your content trained a model without asking" fight lives. Agent is a quieter, third category — and it's the one that fires every time your tool calls out to fetch a page on a user's behalf, whether that's a coding agent following a link in an issue, a research agent pulling a source, or a browser-use agent clicking through a site.

What actually changed on September 15

Three things changed for Agent traffic specifically¹:

1. New ad-monetized domains get a default. At onboarding, a site owner who selects "I monetize pages that serve ads" now gets Training set to Disallow AI Training and Agent set to Block on pages with ads, by default. A site that doesn't monetize with ads still defaults to Allow across all three¹.
2. Existing domains get migrated based on their old setting. A domain that previously had the legacy "Block AI Bots" toggle on — whether "Block" or "Block on pages with ads" — has its Agent control migrated to Block on pages with ads. A domain that had AI blocking off keeps Agent on Allow¹.
3. There's still no "Disallow" option for Agent. Training has three real options (Allow, Disallow AI Training, Block) plus the ads-scoped variant; Search and Agent only have Allow, Block on pages with ads, or Block everywhere. Cloudflare's stated reason: "Agents do not create the same search-discoverability tradeoff as mixed-use crawlers, and the Internet does not yet have a well-established directive for expressing Disallow preferences to agents"¹.

The mechanism that keeps all of this honest is Bot Preference Sync, announced August 21, 2026: whatever you set for Search, Agent, and Training in the dashboard gets mirrored into your robots.txt automatically, so the file you publish and the rule Cloudflare enforces at the edge don't drift apart³. New customers get Sync turned on by default³.

Practically: if you're running an agent that fetches arbitrary URLs, a meaningful and growing slice of the ad-supported web now blocks that fetch by default, with no way for the site to say "search yes, agent no" via robots.txt alone the way it can for training — the enforcement is at the edge, and the file may or may not say so explicitly.

Which bots are "Agent" bots today

Cloudflare's own bot reference table doesn't use the words "Search / Agent / Training" as column values — it labels each crawler with a legacy category (AI Crawler, AI Search, AI Assistant, Search Engine)⁴. Cross-referencing that against the plain-language behavior definitions above: Cloudflare's own definition for AI Assistant reads "Automated AI bot driven by user action"² — which is exactly how Cloudflare describes Agent behavior. The real, Verified bots in that category today⁴:

These are distinct from each operator's training crawler (GPTBot, ClaudeBot) and search crawler (OAI-SearchBot, Claude-SearchBot, PerplexityBot) — three separate, separately-blockable bots per major operator⁴.

If your own agent isn't one of these, none of the above applies to you by name. These presets and migrations govern the specific, tracked crawlers in Cloudflare's directory — each one self-identifying with its own distinct User-Agent, the way the table above shows⁴. A custom agent using its own descriptive User-Agent (like the one this post builds below) isn't one of those named entries, so the Agent preset doesn't single it out directly; it's evaluated by whatever general Bot Management rules the site already has configured, which historically treat unrecognized automated traffic with more suspicion, not less². Formally joining Cloudflare's Verified Bot directory — via a cryptographic Web Bot Auth signature or a confirmed, stable IP range, plus a track record of honest, non-abusive behavior — is what additionally earns a bot the kind of good-faith treatment the "Accountable" operators get, and is the more durable path for an agent that fetches at any real scale². We covered how a bot earns that status — and why the requirement is shifting from IP allowlists toward cryptographic signing — in our Web Bot Auth deep dive.

Building a web-fetch tool that handles this correctly

The wrong reaction to a 403 is to retry with a browser User-Agent and hope nobody notices. That's the exact failure mode Cloudflare's "Accountable" designation and Verified Bot program exist to squeeze out of the ecosystem — an operator that behaves dishonestly when blocked is disqualified from the good-faith treatment those programs extend¹². The right reaction is to identify honestly, check the site's stated preference first, and treat different HTTP outcomes differently.

Step 1: An honest, self-describing User-Agent

Don't impersonate ChatGPT-User or Claude-User — those tokens are cryptographically or IP-verified to specific operators, and presenting them from your own code is exactly the "not honest about who it is" behavior that gets a bot delisted². Use a descriptive string that identifies your own agent instead:

USER_AGENT = "NerdLevelTechAgent/1.0 (+https://nerdleveltech.com/agent-info; contact=agents@nerdleveltech.com)"

This alone won't get you Verified-bot treatment — that requires applying to Cloudflare's program and implementing Web Bot Auth or IP validation² — but it's the honest baseline every other step depends on.

Step 2: Check robots.txt before you fetch

Python's standard library already ships a robots.txt parser — there's no reason to hand-roll Disallow parsing. Worth knowing its limits, though: urllib.robotparser predates RFC 9309, the IETF Robots Exclusion Protocol standard⁵, and doesn't implement the RFC's longest-match-wins rule for conflicting rules — it goes with whichever line comes first in the file instead. For the one behavior this post leans on — what happens when robots.txt can't be fetched at all — its default lines up with the RFC:

from urllib.robotparser import RobotFileParser
from urllib.parse import urljoin

def robots_for(base_url: str) -> RobotFileParser:
    rp = RobotFileParser()
    rp.set_url(urljoin(base_url, "/robots.txt"))
    try:
        rp.read()
    except Exception:
        pass  # can't reach the site at all -> parser stays "unread"; can_fetch()
              # then defaults to disallow, which matches RFC 9309 for this case
    return rp

If a site owner has published a Disallow line matching your declared agent name — whether by hand or via Bot Preference Sync — can_fetch() catches it before you ever send the real request.

Step 3: Treat 403 and 402 as different outcomes

AI Crawl Control, the Cloudflare product behind these settings and available on every plan⁶, lets a site owner configure the block response as either 403 Forbidden ("you may not have this") or 402 Payment Required ("you may have this if you pay") — that's a deliberate choice Cloudflare's dashboard exposes, aimed at giving a crawler operator a path from "blocked" to "licensed"⁷. A tool that treats both as one generic failure throws that signal away:

from dataclasses import dataclass
from urllib import request, error
import json

USER_AGENT = "NerdLevelTechAgent/1.0 (+https://nerdleveltech.com/agent-info; contact=agents@nerdleveltech.com)"

@dataclass
class FetchResult:
    status: str   # "ok" | "disallowed" | "blocked" | "payment_required" | "error"
    url: str
    detail: str = ""

class PoliteAgentFetcher:
    def __init__(self, user_agent: str = USER_AGENT):
        self.user_agent = user_agent
        self._robots_cache: dict[str, RobotFileParser] = {}

    def _robots_for(self, base_url: str) -> RobotFileParser:
        if base_url not in self._robots_cache:
            self._robots_cache[base_url] = robots_for(base_url)
        return self._robots_cache[base_url]

    def fetch(self, url: str, base_url: str) -> FetchResult:
        robots = self._robots_for(base_url)
        if not robots.can_fetch(self.user_agent, url):
            if not robots.last_checked:
                return FetchResult("disallowed", url, "robots.txt was never successfully read -- defaulting to disallow, per RFC 9309")
            return FetchResult("disallowed", url, "robots.txt Disallow matches our own declared agent name")

        req = request.Request(url, headers={"User-Agent": self.user_agent})
        try:
            with request.urlopen(req, timeout=5) as resp:
                return FetchResult("ok", url, resp.read().decode(errors="replace")[:80])
        except error.HTTPError as e:
            if e.code == 403:
                return FetchResult("blocked", url, "403 -- Agent behavior is set to Block here. Do not retry with a different User-Agent.")
            if e.code == 402:
                body = e.read().decode(errors="replace")
                info_url = json.loads(body).get("info_url", "") if body else ""
                return FetchResult("payment_required", url, f"402 -- a paid access path may exist: {info_url}")
            return FetchResult("error", url, f"HTTP {e.code}")
        except Exception as e:
            return FetchResult("error", url, str(e))

Feed the payment_required case back to whatever layer of your agent decides whether to spend money — a Pay Per Crawl integration, a human approval step, or simply "skip this source" — rather than silently dropping it or treating it the same as a hard 403⁷⁸.

Running it

To see all four outcomes without depending on a live, real-world Cloudflare zone, a minimal local server reproduces the two response codes AI Crawl Control's own dashboard lets a site owner choose between⁷, plus a Bot Preference Sync-style robots.txt³:

$ python3 demo.py
/                    -> ok                 <html><body>Ordinary page content.</body></html>
/internal/secret     -> disallowed         robots.txt Disallow matches our own declared agent name
/blocked             -> blocked            403 -- Agent behavior is set to Block here. Do not retry with a different User-Agent.
/paywalled           -> payment_required   402 -- a paid access path may exist: https://example.com/pay-per-crawl

Four distinct, correctly-classified outcomes from four requests against the mock server — none of which required guessing at what a real blocked response looks like, because the status codes and the robots.txt-sync mechanism are both documented, not observed.

There's a fifth outcome worth testing separately: what happens when robots.txt can't be reached at all — DNS failure, connection refused, timeout — rather than returning a normal HTTP response. Pointed at a host that doesn't resolve, the same fetcher returns:

(unreachable robots.txt) -> disallowed   robots.txt was never successfully read -- defaulting to disallow, per RFC 9309

That's the last_checked branch added above. can_fetch() degrades to "disallow everything" once the parser never manages to read a file, so the fetch tool reports that case as disallowed with a detail message that says why — instead of reusing the wording for an actual Disallow line and hiding the difference from whoever debugs the run later. The message deliberately doesn't say "unreachable" — last_checked stays unset both for a genuine network failure and for the 401/403-on-robots.txt gotcha described below, and RobotFileParser doesn't expose which of the two actually happened.

What not to do

Don't retry a 403 with a different User-Agent, a headless browser, or a residential proxy to make the request look human. Cloudflare's entire "Accountable" framework is built around rewarding operators who are transparent about their identity and behavior, and around a Verified Bot program that can and does delist bots caught misrepresenting themselves¹². An agent that spoofs its way past a block isn't solving the block — it's providing the exact evidence Cloudflare's transparency requirements are designed to catch, and it undermines the case for every other agent operator asking to be treated as Accountable.

If a source consistently blocks your agent and you need it, the paths that don't involve deception are: apply for Verified Bot status and implement Web Bot Auth²⁹, reach out to the site owner directly, or use Pay Per Crawl / Pay Per Use where the operator supports it⁷⁸.

Production patterns worth copying

Cache the robots.txt parse per domain, not per request. RobotFileParser objects are cheap to reuse; refetching robots.txt on every URL from the same site adds latency and traffic for no benefit. The _robots_cache dict above is the minimum version of this.

Log disallowed, blocked, and payment_required differently in your agent's trace. They mean different things to whoever is debugging a run later: "the site said no in writing," "the edge said no at request time," and "the site has a price," respectively. Collapsing them into one "fetch failed" line throws away information you'll want during a postmortem.

Don't conflate "missing" with "unreachable." RFC 9309 treats these as opposite cases⁵: a 404 on /robots.txt is its "Unavailable" status, and a crawler MAY treat that as full access — a 404 is not a Disallow: /. A robots.txt fetch that fails outright (DNS failure, connection refused, timeout, a 5xx) is "Unreachable," and the RFC says a crawler MUST assume complete disallow until the file can actually be read. RobotFileParser matches that second rule: if .read() never completes, can_fetch() defaults to blocking everything — which is exactly why the last_checked check above matters, since it's the only way to tell "the site said no" apart from "the request never got an answer." One more wrinkle worth knowing: RobotFileParser also treats a 401 or 403 while fetching robots.txt itself as full disallow — a Python-specific choice, not something RFC 9309 mandates for that status code.

Verification

The Python code above runs as shown — every snippet was executed against a local http.server-based mock, not a live Cloudflare-protected domain. This sandbox's outbound network access is restricted to an allowlist that doesn't include arbitrary third-party sites, so a real request against a Cloudflare zone with Agent set to Block wasn't possible while writing this post.

What the mock does and doesn't prove: it confirms the fetch tool correctly branches on 200, 403, 402, and a robots.txt Disallow line, because those are the four response shapes Cloudflare's own documentation specifies AI Crawl Control can produce — 403 Forbidden and 402 Payment Required as the two configurable block-response codes⁷, and a robots.txt entry synced from an Agent-category block or disallow choice³. It does not prove what the response body or headers look like on a real, currently-configured Cloudflare zone, since that content is set per site owner and isn't published as a fixed spec. Every fact about the policy itself — the three behaviors, the September 15 defaults, the migration table, the bot names and User-Agent strings — is cited below to Cloudflare's own blog and developer docs, fetched September 16, 2026.

The bottom line

The mixed-use crawler fix is the part of Cloudflare's September 15 announcement that made headlines, because it's the part that resolves a genuine, years-old fight between publishers and search engines. But if you build agents rather than publish pages, the quieter change is the one that affects your code: Agent is now its own bot behavior, with its own default, on a growing share of the ad-supported web. The fix on your side isn't a workaround — it's a fetch tool that says who it is, reads what the site published, and treats a 403 and a 402 as the different signals they actually are.

Related reads

• Web Bot Auth in 2026: Shipped Before It's a Standard
• AI Agent Payments: The x402 Standards Race in 2026
• Web Search for AI Agents: Prices and Indexes in 2026