AI crawler bots and robots.txt reference

The short answer

There is no single AI crawler switch. Search discovery, model training, user-triggered retrieval, product controls, and ad validation are different jobs, even when one provider operates all of them. OpenAI currently names OAI-SearchBot, GPTBot, ChatGPT-User, and OAI-AdsBot. Anthropic names Claude-SearchBot, ClaudeBot, and Claude-User. Perplexity separates PerplexityBot from Perplexity-User. Google Search uses Googlebot, while Google-Extended is a policy token with no HTTP user agent of its own.^1,2,3,4

Choose by purpose. A common publisher policy is to allow automatic search crawlers, keep user-requested fetches reachable where the provider honours that choice, and decline training crawlers. That preserves a route into live answers without granting every use of the content. The important exception is Google-Extended: Google currently couples future Gemini training and Gemini grounding under the same token, so blocking it also removes a route into grounded Gemini answers while leaving Google Search and AI Overviews under Googlebot.

AI crawler reference: every named provider token

This table is generated from the same crawler registry used by Robot Visible's exact-URL policy checks. Roles are deliberately separate. A training bot visit must not be reported as search visibility, a user-triggered fetch is not an automatic crawl, and Google-Extended must not appear in a server-log checklist because it never sends a request.

A copy-paste robots.txt policy, with the trade-offs

The example below allows the automatic crawlers that build search and answer indexes while declining the named training collectors. It is a starting policy, not a universal recommendation. Put it in the robots.txt file at the root of every hostname it should govern, preserve any existing path-specific rules, and test the exact public URL after publishing. Provider changes may take time to propagate.

Read every line as a product choice. Allowing OAI-SearchBot, PerplexityBot, and Claude-SearchBot keeps those automatic discovery paths open; it does not guarantee retrieval or citation. Allowing Googlebot preserves Google Search and its AI features, but indexing remains a separate decision. Blocking GPTBot and ClaudeBot declines their future training collection without blocking their providers' separate search bots. Blocking Google-Extended declines both future Gemini training and Gemini grounding, because Google does not currently expose separate tokens for those uses.

# ChatGPT search: allow discovery for ChatGPT search answers
User-agent: OAI-SearchBot
Allow: /

# OpenAI model training: decline without blocking ChatGPT search
User-agent: GPTBot
Disallow: /

# Google Search, AI Overviews and AI Mode: keep Googlebot eligible
User-agent: Googlebot
Allow: /

# Gemini training and Gemini grounding: one coupled Google control
User-agent: Google-Extended
Disallow: /

# Perplexity automatic search discovery
User-agent: PerplexityBot
Allow: /

# Claude automatic search discovery
User-agent: Claude-SearchBot
Allow: /

# Anthropic model training: decline without blocking Claude search
User-agent: ClaudeBot
Disallow: /

What robots.txt can honestly establish

A robots.txt check can establish whether the fetched policy permits a named product token to fetch one exact URL at that moment. It does not prove that the provider visited, received the same response through a firewall, rendered JavaScript, indexed the page, retrieved it for a question, or cited it. A missing or failed robots.txt fetch is an unknown observation, not a confident allow.

The distinction matters at the WAF layer. A policy can say Allow while a CDN challenge, rate limit, authentication wall, or IP rule still refuses the real provider. The inverse also happens: a claimed bot can ignore robots.txt or spoof a familiar name. Treat policy and observed requests as two evidence rows rather than collapsing them into a green crawler score.

How to verify an AI crawler by IP

A user-agent can be spoofed, so the string in a log is only a claim. For OpenAI, Perplexity, and Anthropic, match both the product token and the source address against the provider's current JSON list. Google supports published IP ranges and forward-confirmed reverse DNS: the hostname must end in an official Google crawler suffix, and resolving that hostname must return the original address. Never copy individual addresses into an article or a permanent WAF rule when the provider publishes a maintained endpoint.^1,2,3,4,5

For an observed visit, retain the requested path, timestamp, full user-agent, source IP, response status, bytes served, and any WAF or challenge action. That proves a request reached your infrastructure and what your infrastructure returned. It still does not prove how the provider later indexed, interpreted, or used the response.

Content Signals and the Content-Usage direction

Cloudflare's Content Signals convention adds search, ai-input, and ai-train preferences to robots.txt, and its newer content-use experiment distinguishes immediate interaction, reference-style indexing, and fuller reuse. Cloudflare can parse and enforce these signals in its own products, but the convention is not a substitute for provider-specific groups: no major answer provider has publicly committed to treating Cloudflare's vocabulary as its publisher control.⁶

The standards-track direction is the IETF AI Preferences work. Its active vocabulary draft describes allow, disallow, and unknown preferences; a companion attachment draft proposed a Content-Usage HTTP header and robots.txt directive. As of 10 August 2026 the vocabulary is still an Internet-Draft, the attachment draft has expired, and the work is not yet an RFC. That makes Content-Usage useful to watch and unsuitable to present as a dependable production opt-out today.^7,8

If you publish an experimental signal, keep the ordinary provider rules beside it and document which enforcement layer actually reads it. Unknown fields are normally ignored by robots parsers; that is not the same thing as a provider accepting the preference. The safe operational posture is layered: explicit provider tokens for declared choices, current WAF rules for enforcement, and emerging usage signals as additional metadata rather than a replacement.

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /

What to check before Cloudflare changes its defaults

Cloudflare's new defaults take effect on 15 September 2026 for specified customers and ad-serving pages. Before then, record the intended policy for search, training, and user-agent traffic; inspect the dashboard's migrated settings; fetch the public robots.txt; and test the important URLs from outside your authenticated session. The companion Cloudflare deadline guide separates who is affected from who keeps an existing configuration.⁶

After any change, check both layers again. First establish the exact-URL policy from robots.txt. Then inspect production logs for verified provider traffic and the response it received. Neither layer licenses a visibility claim: only a captured answer with the question, provider, surface, cited URL, and timestamp establishes an observed citation.

Run a free exact-URL crawler policy check

How this reference is maintained

Every crawler row is derived from the scanner registry and was checked against first-party documentation on 2026-08-10. The build fails when the oldest review is more than 45 days old. A provider change therefore has one update path: change the registry facts, update the review date and supporting fixture, then let the guide and product read the same record.

The check protects freshness, not truth by itself. Re-verification still requires a person to read each provider page, compare names, roles, robots behaviour, and verification endpoints, and record material changes. A passing date guard means the review happened recently; it does not claim the provider gave advance notice or that every request will comply.