Cloudflare AI crawler defaults explained

What changes on 15 September 2026

On 1 July 2026 Cloudflare announced a new way of classifying automated traffic, and a set of defaults that take effect on 15 September 2026. Instead of treating every non-human request as one category, Cloudflare now separates three behaviours: crawling for search, crawling for model training, and fetching on behalf of a user in real time.

From that date, the default configuration allows search and blocks training and agent traffic on pages that display ads. Search crawlers are not blocked by the new defaults. Site owners can change any of it, in either direction, at any time.

The legacy "Block AI bots" toggle is being deprecated on the same date. If your site uses it, your configuration moves onto the new three-category model, and mixed-purpose crawlers that combine search with training are blocked by it.

Who the new defaults apply to

This is the part most coverage gets wrong in one direction or the other. Cloudflare's press release names three groups, and the distinction matters because only one of them is affected without doing anything.

Search, training and agent are three different crawls

This split is the genuinely useful part of the announcement, and it outlasts the deadline. These three behaviours have always been distinct, and treating them as one category is why so much crawler advice is confused. Blocking "AI bots" as a single class has, for years, also blocked the retrieval that puts a page into an answer.^1,2

Why mixed-use crawlers are treated separately

Some crawlers do more than one of those jobs under a single user agent, so a site owner cannot allow the search behaviour and decline the training behaviour. Cloudflare's term for these is mixed or multi-purpose crawlers, and it evaluates them against all of their behaviours rather than the most permissive one.

Where a crawler does not give the site owner that choice, it is blocked on all pages that display ads. The effect is a commercial one: a crawler that separates its behaviours keeps its search access, and one that bundles them does not. Whether that pressure works is not something anyone can claim yet.

Whether this actually affects you

Work through these in order. Most sites will stop at the first or second line, and that is the honest result rather than a disappointing one.

• Not on Cloudflare? Nothing here applies. The change is a Cloudflare default, not an industry standard.
• No ads on your pages? The training and agent blocks are scoped to pages that display ads, so the new defaults do not change what reaches your content.
• On a paid plan with settings you have already configured? Your configuration is kept.
• On a free plan, displaying ads, and never touched your AI bot settings? This is the case that changes on 15 September without any action from you.
• Using the legacy "Block AI bots" toggle? It is deprecated on 15 September regardless of plan, and it now blocks mixed-purpose crawlers that combine search and training. If you turned it on to stop training, check whether it is also costing you retrieval.

How to check and change your setting

The controls live in your Cloudflare dashboard under Security Settings, in "Configure AI bot policies". Each of the three categories can be set independently, and Cloudflare offers the same three choices for each: block on all pages, block on pages with ads, or allow.

Do this before 15 September if you want a different outcome from the default. Cloudflare has said all customers can opt out of the new defaults at any time before that date, and settings remain editable afterwards.

• Open Security Settings and read your current AI bot configuration before changing anything, so you know what you are changing from.
• Decide the three categories separately. Allowing search while declining training is now expressible, and it was not under a single AI-bots toggle.
• If you rely on assistants retrieving your pages live, look hard at the agent category before leaving it blocked.
• Re-check that your important pages are still reachable after any change, rather than assuming the setting did what you intended.

Run a free scan to see what a crawler can currently read

Pay Per Use: paid when content is used, not when it is fetched

Alongside the defaults, Cloudflare said it is evolving Pay Per Crawl into Pay Per Use. Its stated reasoning is that crawling is a crude measure of value: a page might be crawled once and cited in thousands of answers, or crawled repeatedly and never used.

Two partners are named. With Ceramic.ai, publishers who opt in can be paid when their content appears in search results, and participating publishers get reporting on the queries their content appeared for, the page and snippet used, and average result position. With You.com, an agent can pay on demand for a specific piece of premium content.

Cloudflare describes this as experimental and has not published a general availability date, so treat it as a direction rather than a revenue line. The reporting is the part worth watching: query-level evidence of which page and snippet an answer used is the measurement most publishers currently lack.

What this does not change about AI visibility

A crawler policy decides whether a request is allowed. It does not decide whether your page is worth citing once it arrives. Sites that were not being cited before 15 September will not be cited afterwards because a default changed, and the reverse is also true.

The useful response to a dated platform change is to know your own position rather than react to a headline: what your current setting is, whether your key pages are reachable and readable, and whether anything is actually citing them today. If a change on 15 September moves your evidence, you can only tell that against a baseline recorded before it.

How to track AI visibility over time