---
title: 'Cloudflare Blocks AI Agents: A Tested Fallback (2026)'
description: Cloudflare now blocks AI agents by default on ad-monetized pages. A tested Python pattern for web-fetch tools to detect it and fail gracefully, not spoof it.
url: https://nerdleveltech.com/cloudflare-blocks-ai-agents-fallback-pattern
language: en
date: '2026-09-16'
category: ai-ml
tags:
  - cloudflare
  - ai agents
  - web fetch tool
  - bot management
  - ai crawl control
  - robots.txt
markdown_url: https://nerdleveltech.com/cloudflare-blocks-ai-agents-fallback-pattern.md
attribution: 'Source: NerdLevelTech — https://nerdleveltech.com/cloudflare-blocks-ai-agents-fallback-pattern'
terms: https://nerdleveltech.com/llms.txt
---

# Cloudflare Blocks AI Agents: A Tested Fallback (2026)

Cloudflare now treats AI agents as a separate, blockable category from search and training crawlers. Since September 15, 2026, new ad-monetized sites default to blocking that **Agent** behavior on pages with ads, and existing sites that blocked AI bots before get migrated to the same default automatically[^1].

If your agent has a tool that fetches web pages, that's the change that affects your code, not the "mixed-use crawler" story most coverage led with.

## TL;DR

Cloudflare's Bot Management now recognizes "Agent" as its own bot behavior, distinct from Search and Training: "user-directed agents visiting a page on behalf of a human, such as chat fetch bots and browser-use agents"[^1][^3]. As of September 15, 2026, new ad-monetized domains default to blocking that behavior on pages that serve ads, and existing domains that had the old "Block AI" setting on get migrated the same way[^1]. This post builds and runs a small Python web-fetch tool that identifies itself honestly, checks `robots.txt` before fetching, and treats a `403` and a `402` response as two different, meaningful outcomes — instead of retrying with a different User-Agent, which is exactly the behavior this policy exists to push back on.

## What you'll learn

- What changed for AI agents specifically on September 15, 2026, and how it's different from the "block AI crawlers" story most coverage led with
- Why "Agent" is its own bot behavior, separate from the "mixed-use crawler" (Search + Training) problem the same announcement spent most of its words on
- Which real, Verified bots fall under Agent behavior today, by name and operator
- A tested, dependency-free Python pattern for a web-fetch tool that checks `robots.txt`, identifies honestly, and handles `403` vs. `402` as distinct outcomes
- Why spoofing a Verified bot's User-Agent to dodge a block is a worse idea now than it was a year ago
- What this means practically if you're building your own agent, not shipping one at OpenAI or Anthropic's scale

## Three behaviors, not one "block AI" switch

Most coverage of Cloudflare's September 15 announcement led with "mixed-use crawlers" — a single crawler, like Googlebot, that does both search indexing and AI training, which historically forced site owners into an all-or-nothing choice. Cloudflare's fix for that is a new **Disallow AI Training** setting, and a new **Accountable** designation. Apple, Google, and Microsoft earn it for their mixed-use crawlers by providing opt-outs, transparency, and a guarantee that disallowing training won't hurt search rankings; Amazon, Anthropic, Meta, and OpenAI earn the same designation for their separate, training-only crawlers, since keeping search and training on two different bots means blocking one was never going to touch the other[^1].

That's real, and it's the headline. But it's not the part that matters if you're the one building the agent, rather than the one publishing the pages an agent might fetch.

Since July 1, 2026, Cloudflare has classified bot traffic into three independently controllable behaviors[^1][^2][^3]:

| Behavior | Cloudflare's definition |
|---|---|
| **Search** | Crawling to build search indexes or RAG databases[^3] |
| **Training** | Crawling to train or fine-tune models[^1][^3] |
| **Agent** | User-directed agents visiting a page on behalf of a human, such as chat fetch bots and browser-use agents[^1][^3] |

Search and Training get most of the press because that's where the "your content trained a model without asking" fight lives. Agent is a quieter, third category — and it's the one that fires every time your tool calls out to fetch a page on a user's behalf, whether that's a coding agent following a link in an issue, a research agent pulling a source, or a browser-use agent clicking through a site.

## What actually changed on September 15

Three things changed for Agent traffic specifically[^1]:

1. **New ad-monetized domains get a default.** At onboarding, a site owner who selects "I monetize pages that serve ads" now gets Training set to Disallow AI Training and **Agent set to Block on pages with ads**, by default. A site that doesn't monetize with ads still defaults to Allow across all three[^1].
2. **Existing domains get migrated based on their old setting.** A domain that previously had the legacy "Block AI Bots" toggle on — whether "Block" or "Block on pages with ads" — has its Agent control migrated to **Block on pages with ads**. A domain that had AI blocking off keeps Agent on Allow[^1].
3. **There's still no "Disallow" option for Agent.** Training has three real options (Allow, Disallow AI Training, Block) plus the ads-scoped variant; Search and Agent only have Allow, Block on pages with ads, or Block everywhere. Cloudflare's stated reason: "Agents do not create the same search-discoverability tradeoff as mixed-use crawlers, and the Internet does not yet have a well-established directive for expressing Disallow preferences to agents"[^1].

The mechanism that keeps all of this honest is **Bot Preference Sync**, announced August 21, 2026: whatever you set for Search, Agent, and Training in the dashboard gets mirrored into your `robots.txt` automatically, so the file you publish and the rule Cloudflare enforces at the edge don't drift apart[^2]. New customers get Sync turned on by default[^2].

Practically: if you're running an agent that fetches arbitrary URLs, a meaningful and growing slice of the ad-supported web now blocks that fetch by default, with no way for the site to say "search yes, agent no" via `robots.txt` alone the way it can for training — the enforcement is at the edge, and the file may or may not say so explicitly.

## Which bots are "Agent" bots today

Cloudflare's own bot reference table doesn't use the words "Search / Agent / Training" as column values — it labels each crawler with a legacy **category** (`AI Crawler`, `AI Search`, `AI Assistant`, `Search Engine`)[^4]. Cross-referencing that against the plain-language behavior definitions above: Cloudflare's own definition for `AI Assistant` reads "Automated AI bot driven by user action"[^3] — which is exactly how Cloudflare describes Agent behavior. The real, Verified bots in that category today[^4]:

| Crawler | Operator | User-Agent |
|---|---|---|
| ChatGPT-User | OpenAI | `ChatGPT-User` |
| Claude-User | Anthropic | `Claude-User` |
| Perplexity-User | Perplexity | `Perplexity-User` |
| Meta-ExternalFetcher | Meta | `meta-externalfetcher` |
| DuckAssistBot | DuckDuckGo | `DuckAssistBot` |
| MistralAI-User | Mistral | `MistralAI-User` |

These are distinct from each operator's *training* crawler (`GPTBot`, `ClaudeBot`) and *search* crawler (`OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot`) — three separate, separately-blockable bots per major operator[^4].

If your own agent isn't one of these, none of the above applies to you by name. These presets and migrations govern the specific, tracked crawlers in Cloudflare's directory — each one self-identifying with its own distinct User-Agent, the way the table above shows[^4]. A custom agent using its own descriptive User-Agent (like the one this post builds below) isn't one of those named entries, so the Agent preset doesn't single it out directly; it's evaluated by whatever general Bot Management rules the site already has configured, which historically treat unrecognized automated traffic with more suspicion, not less[^3]. Formally joining Cloudflare's Verified Bot directory — via a cryptographic Web Bot Auth signature or a confirmed, stable IP range, plus a track record of honest, non-abusive behavior — is what additionally earns a bot the kind of good-faith treatment the "Accountable" operators get, and is the more durable path for an agent that fetches at any real scale[^3]. We covered how a bot earns that status — and why the requirement is shifting from IP allowlists toward cryptographic signing — in [our Web Bot Auth deep dive](https://nerdleveltech.com/web-bot-auth-ietf-standard-agent-verification).

## Building a web-fetch tool that handles this correctly

The wrong reaction to a `403` is to retry with a browser User-Agent and hope nobody notices. That's the exact failure mode Cloudflare's "Accountable" designation and Verified Bot program exist to squeeze out of the ecosystem — an operator that behaves dishonestly when blocked is disqualified from the good-faith treatment those programs extend[^1][^3]. The right reaction is to identify honestly, check the site's stated preference first, and treat different HTTP outcomes differently.

### Step 1: An honest, self-describing User-Agent

Don't impersonate `ChatGPT-User` or `Claude-User` — those tokens are cryptographically or IP-verified to specific operators, and presenting them from your own code is exactly the "not honest about who it is" behavior that gets a bot delisted[^3]. Use a descriptive string that identifies your own agent instead:

```python
USER_AGENT = "NerdLevelTechAgent/1.0 (+https://nerdleveltech.com/agent-info; contact=agents@nerdleveltech.com)"
```

This alone won't get you Verified-bot treatment — that requires applying to Cloudflare's program and implementing Web Bot Auth or IP validation[^3] — but it's the honest baseline every other step depends on.

### Step 2: Check `robots.txt` before you fetch

Python's standard library already ships a `robots.txt` parser — there's no reason to hand-roll `Disallow` parsing. Worth knowing its limits, though: `urllib.robotparser` predates RFC 9309, the IETF Robots Exclusion Protocol standard[^7], and doesn't implement the RFC's longest-match-wins rule for conflicting rules — it goes with whichever line comes first in the file instead. For the one behavior this post leans on — what happens when `robots.txt` can't be fetched at all — its default lines up with the RFC:

```python
from urllib.robotparser import RobotFileParser
from urllib.parse import urljoin

def robots_for(base_url: str) -> RobotFileParser:
    rp = RobotFileParser()
    rp.set_url(urljoin(base_url, "/robots.txt"))
    try:
        rp.read()
    except Exception:
        pass  # can't reach the site at all -> parser stays "unread"; can_fetch()
              # then defaults to disallow, which matches RFC 9309 for this case
    return rp
```

If a site owner has published a `Disallow` line matching your declared agent name — whether by hand or via Bot Preference Sync — `can_fetch()` catches it before you ever send the real request.

### Step 3: Treat `403` and `402` as different outcomes

**AI Crawl Control**, the Cloudflare product behind these settings and available on every plan[^6], lets a site owner configure the block response as either `403 Forbidden` ("you may not have this") or `402 Payment Required` ("you may have this if you pay") — that's a deliberate choice Cloudflare's dashboard exposes, aimed at giving a crawler operator a path from "blocked" to "licensed"[^5]. A tool that treats both as one generic failure throws that signal away:

```python
from dataclasses import dataclass
from urllib import request, error
import json

USER_AGENT = "NerdLevelTechAgent/1.0 (+https://nerdleveltech.com/agent-info; contact=agents@nerdleveltech.com)"

@dataclass
class FetchResult:
    status: str   # "ok" | "disallowed" | "blocked" | "payment_required" | "error"
    url: str
    detail: str = ""

class PoliteAgentFetcher:
    def __init__(self, user_agent: str = USER_AGENT):
        self.user_agent = user_agent
        self._robots_cache: dict[str, RobotFileParser] = {}

    def _robots_for(self, base_url: str) -> RobotFileParser:
        if base_url not in self._robots_cache:
            self._robots_cache[base_url] = robots_for(base_url)
        return self._robots_cache[base_url]

    def fetch(self, url: str, base_url: str) -> FetchResult:
        robots = self._robots_for(base_url)
        if not robots.can_fetch(self.user_agent, url):
            if not robots.last_checked:
                return FetchResult("disallowed", url, "robots.txt was never successfully read -- defaulting to disallow, per RFC 9309")
            return FetchResult("disallowed", url, "robots.txt Disallow matches our own declared agent name")

        req = request.Request(url, headers={"User-Agent": self.user_agent})
        try:
            with request.urlopen(req, timeout=5) as resp:
                return FetchResult("ok", url, resp.read().decode(errors="replace")[:80])
        except error.HTTPError as e:
            if e.code == 403:
                return FetchResult("blocked", url, "403 -- Agent behavior is set to Block here. Do not retry with a different User-Agent.")
            if e.code == 402:
                body = e.read().decode(errors="replace")
                info_url = json.loads(body).get("info_url", "") if body else ""
                return FetchResult("payment_required", url, f"402 -- a paid access path may exist: {info_url}")
            return FetchResult("error", url, f"HTTP {e.code}")
        except Exception as e:
            return FetchResult("error", url, str(e))
```

Feed the `payment_required` case back to whatever layer of your agent decides whether to spend money — a Pay Per Crawl integration, a human approval step, or simply "skip this source" — rather than silently dropping it or treating it the same as a hard `403`[^5][^9].

### Running it

To see all four outcomes without depending on a live, real-world Cloudflare zone, a minimal local server reproduces the two response codes AI Crawl Control's own dashboard lets a site owner choose between[^5], plus a Bot Preference Sync-style `robots.txt`[^2]:

```
$ python3 demo.py
/                    -> ok                 <html><body>Ordinary page content.</body></html>
/internal/secret     -> disallowed         robots.txt Disallow matches our own declared agent name
/blocked             -> blocked            403 -- Agent behavior is set to Block here. Do not retry with a different User-Agent.
/paywalled           -> payment_required   402 -- a paid access path may exist: https://example.com/pay-per-crawl
```

Four distinct, correctly-classified outcomes from four requests against the mock server — none of which required guessing at what a real blocked response looks like, because the status codes and the `robots.txt`-sync mechanism are both documented, not observed.

There's a fifth outcome worth testing separately: what happens when `robots.txt` can't be reached at all — DNS failure, connection refused, timeout — rather than returning a normal HTTP response. Pointed at a host that doesn't resolve, the same fetcher returns:

```
(unreachable robots.txt) -> disallowed   robots.txt was never successfully read -- defaulting to disallow, per RFC 9309
```

That's the `last_checked` branch added above. `can_fetch()` degrades to "disallow everything" once the parser never manages to read a file, so the fetch tool reports that case as `disallowed` with a detail message that says why — instead of reusing the wording for an actual `Disallow` line and hiding the difference from whoever debugs the run later. The message deliberately doesn't say "unreachable" — `last_checked` stays unset both for a genuine network failure and for the 401/403-on-`robots.txt` gotcha described below, and `RobotFileParser` doesn't expose which of the two actually happened.

## What not to do

Don't retry a `403` with a different User-Agent, a headless browser, or a residential proxy to make the request look human. Cloudflare's entire "Accountable" framework is built around rewarding operators who are transparent about their identity and behavior, and around a Verified Bot program that can and does delist bots caught misrepresenting themselves[^1][^3]. An agent that spoofs its way past a block isn't solving the block — it's providing the exact evidence Cloudflare's transparency requirements are designed to catch, and it undermines the case for every other agent operator asking to be treated as Accountable.

If a source consistently blocks your agent and you need it, the paths that don't involve deception are: apply for Verified Bot status and implement Web Bot Auth[^3][^8], reach out to the site owner directly, or use Pay Per Crawl / Pay Per Use where the operator supports it[^5][^9].

## Production patterns worth copying

**Cache the `robots.txt` parse per domain, not per request.** `RobotFileParser` objects are cheap to reuse; refetching `robots.txt` on every URL from the same site adds latency and traffic for no benefit. The `_robots_cache` dict above is the minimum version of this.

**Log `disallowed`, `blocked`, and `payment_required` differently in your agent's trace.** They mean different things to whoever is debugging a run later: "the site said no in writing," "the edge said no at request time," and "the site has a price," respectively. Collapsing them into one "fetch failed" line throws away information you'll want during a postmortem.

**Don't conflate "missing" with "unreachable."** RFC 9309 treats these as opposite cases[^7]: a `404` on `/robots.txt` is its "Unavailable" status, and a crawler MAY treat that as full access — a `404` is not a `Disallow: /`. A `robots.txt` fetch that fails outright (DNS failure, connection refused, timeout, a 5xx) is "Unreachable," and the RFC says a crawler MUST assume complete disallow until the file can actually be read. `RobotFileParser` matches that second rule: if `.read()` never completes, `can_fetch()` defaults to blocking everything — which is exactly why the `last_checked` check above matters, since it's the only way to tell "the site said no" apart from "the request never got an answer." One more wrinkle worth knowing: `RobotFileParser` also treats a `401` or `403` while fetching `robots.txt` itself as full disallow — a Python-specific choice, not something RFC 9309 mandates for that status code.

## Verification

The Python code above runs as shown — every snippet was executed against a local `http.server`-based mock, not a live Cloudflare-protected domain. This sandbox's outbound network access is restricted to an allowlist that doesn't include arbitrary third-party sites, so a real request against a Cloudflare zone with Agent set to Block wasn't possible while writing this post.

What the mock does and doesn't prove: it confirms the fetch tool correctly branches on `200`, `403`, `402`, and a `robots.txt` `Disallow` line, because those are the four response shapes Cloudflare's own documentation specifies AI Crawl Control can produce — `403 Forbidden` and `402 Payment Required` as the two configurable block-response codes[^5], and a `robots.txt` entry synced from an Agent-category block or disallow choice[^2]. It does not prove what the response body or headers look like on a real, currently-configured Cloudflare zone, since that content is set per site owner and isn't published as a fixed spec. Every fact about the policy itself — the three behaviors, the September 15 defaults, the migration table, the bot names and User-Agent strings — is cited below to Cloudflare's own blog and developer docs, fetched September 16, 2026.

## FAQ

**Does this affect every AI agent that fetches web pages?**
Only Cloudflare-protected sites, and within those, only for bots on Cloudflare's Verified list under Agent behavior (`ChatGPT-User`, `Claude-User`, `Perplexity-User`, and similar)[^1][^4]. An unverified, custom-built agent isn't covered by these specific presets — it's subject to whatever general Bot Management rules the site already has[^3].

**Is Agent traffic blocked everywhere now, or just on some pages?**
Only on pages Cloudflare detects as serving ads, for both the new-domain default and the existing-domain migration described above. A site can still choose to block Agent everywhere, but that's not the new default[^1].

**Can a site allow Search but block Agent?**
Yes — Search, Agent, and Training are controlled independently. A site can, for example, stay fully indexed while blocking every real-time agent fetch on ad-serving pages[^1][^3].

**Is there a way for a site to say "no agents" through `robots.txt` alone, the way Disallow AI Training works for training?**
Not yet. Cloudflare states explicitly that there's no Disallow-style, self-declared preference for Agent traffic yet, because no equivalent web standard exists for it the way there is for training opt-outs[^1].

**What's the difference between a 403 and a 402 from a Cloudflare-protected site?**
Both are configurable in AI Crawl Control's dashboard. `403` means the site owner does not want the crawler to access the content at all; `402` signals a paid-access path may exist[^5].

## The bottom line

The mixed-use crawler fix is the part of Cloudflare's September 15 announcement that made headlines, because it's the part that resolves a genuine, years-old fight between publishers and search engines. But if you build agents rather than publish pages, the quieter change is the one that affects your code: Agent is now its own bot behavior, with its own default, on a growing share of the ad-supported web. The fix on your side isn't a workaround — it's a fetch tool that says who it is, reads what the site published, and treats a `403` and a `402` as the different signals they actually are.

## Related reads

- [Web Bot Auth in 2026: Shipped Before It's a Standard](https://nerdleveltech.com/web-bot-auth-ietf-standard-agent-verification)
- [AI Agent Payments: The x402 Standards Race in 2026](https://nerdleveltech.com/ai-agent-payments-x402-standards)
- [Web Search for AI Agents: Prices and Indexes in 2026](https://nerdleveltech.com/web-search-for-ai-agents-prices-indexes)

[^1]: Cloudflare, "Have it both ways: stay discoverable in search while disallowing AI training" — https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/ (published September 15, 2026; Search/Agent/Training definitions, Accountable designation and requirements, "what changed on September 15" list, existing- and new-domain migration tables, Applebot/Googlebot/Bingbot specifics, "no Disallow setting for Agents" statement; fetched 2026-09-16)
[^2]: Cloudflare, "Say it once: Introducing Bot Preference Sync" — https://blog.cloudflare.com/bot-preference-sync/ (published August 21, 2026; confirms the July 1, 2026 origin of the Search/Agent/Training taxonomy, Bot Preference Sync mechanics and robots.txt example, publisher vs. non-publisher onboarding defaults, Accountable transparency requirements for mixed-use bots; fetched 2026-09-16)
[^3]: Cloudflare Docs, "Verified bots" — https://developers.cloudflare.com/bots/concepts/bot/verified-bots/ (last updated July 1, 2026; Search/Agent/Training/Transact/etc. behavior table and definitions, Direct vs. Intermediary bot operation, legacy category definitions including AI Assistant/AI Crawler/AI Search, Verified bot honesty/non-abuse requirements and delisting conditions; fetched 2026-09-16)
[^4]: Cloudflare Docs, "Bot reference" — https://developers.cloudflare.com/ai-crawl-control/reference/bots/ (last updated April 23, 2026; per-crawler operator, category, and User-Agent table for GPTBot/ChatGPT-User/OAI-SearchBot, ClaudeBot/Claude-User/Claude-SearchBot, PerplexityBot/Perplexity-User, Meta-ExternalFetcher, DuckAssistBot, MistralAI-User, and others; fetched 2026-09-16)
[^5]: Cloudflare Docs, "Manage AI crawlers" — https://developers.cloudflare.com/ai-crawl-control/features/manage-ai-crawlers/ (last updated July 28, 2026; configurable 403 Forbidden vs. 402 Payment Required block response codes, Pay Per Crawl closed-beta status; fetched 2026-09-16)
[^6]: Cloudflare Docs, "AI Crawl Control" overview — https://developers.cloudflare.com/ai-crawl-control/ (last updated August 14, 2026; product overview, available on all Cloudflare plans; fetched 2026-09-16)
[^7]: RFC 9309, "Robots Exclusion Protocol" — https://www.rfc-editor.org/rfc/rfc9309.html (IETF Standards Track, published September 2022; Section 2.3.1.3 "Unavailable Status" — a 4xx like 404, crawler MAY access anything; Section 2.3.1.4 "Unreachable Status" — network or server errors, crawler MUST assume complete disallow; fetched 2026-09-16)
[^8]: NerdLevelTech, "Web Bot Auth in 2026: Shipped Before It's a Standard" — /web-bot-auth-ietf-standard-agent-verification (2026-08-12; how a bot becomes cryptographically Verified with Cloudflare)
[^9]: Cloudflare Developers Changelog, "Introducing Pay Per Crawl (private beta)" — https://developers.cloudflare.com/changelog/post/2025-07-01-pay-per-crawl/ (private beta launch dated July 1, 2025)