Claude search, citations, and crawler roles

The short answer

Claude search has no publisher submission form or special citation markup. Anthropic documents three web robots with different purposes: ClaudeBot for potential model-training collection, Claude-User for retrieval triggered by a user's request, and Claude-SearchBot for search indexing and answer relevance. A publisher can choose different robots.txt rules for each role; an all-or-nothing ‘AI bot’ rule gives up that control.¹

When Claude uses Anthropic's web-search tool, it can run one or more searches, receive result content and metadata, and answer with source citations. Citations are always enabled for this tool. That documented pipeline establishes how the API surface can expose attribution, but it does not publish a deterministic ranking formula, a complete index status, or a promise that the same sources will appear in claude.ai.²

Keep three evidence states separate. Eligible evidence shows that the correct public URL can be discovered or retrieved under the intended policy. Competitive evidence compares the page's claims, freshness, authority, and usefulness with the sources Claude actually selects. Observed evidence preserves a particular answer, source URL, query, provider, surface, and timestamp. One state never licenses a claim about another. A crawler request is not a citation, and a citation is not proof that a specific crawler caused it.

Claude's three crawler roles are independent

Anthropic's crawler documentation separates the roles explicitly. ClaudeBot collects web content that may contribute to model development. Claude-User reaches pages as part of a user-initiated request. Claude-SearchBot indexes and analyses web content to improve the relevance and accuracy of search results. The names describe intended uses, not three stages that every cited page must pass through in order.¹

Anthropic says all three honor robots.txt disallow signals and that rules should be configured independently. It also warns that blocking Claude-User can prevent retrieval for user queries and may reduce visibility in user-directed search, while blocking Claude-SearchBot may reduce visibility and accuracy in search results. Blocking ClaudeBot signals that future collected material should not be included in Anthropic's model-training datasets; it is not the documented switch for search inclusion.¹

How Claude web search returns sources and citations

Anthropic's API web-search flow is agentic rather than a single fixed lookup: Claude decides whether to search, the API executes it, and Claude may search again before producing the final response. Publishers should therefore expect the number and wording of searches—and the final source set—to vary even when the visible user question is unchanged. The documented `max_uses` parameter limits search operations, not sources or citations.²

A returned web-search result can include a URL, title, page age, and encrypted content. A final citation annotation can include the cited URL, title, encrypted index, and a cited-text excerpt of up to 150 characters. Those are useful observation fields: the search result says content entered the tool context, while a citation annotation says the final API answer attributed a passage to that URL. Neither field reveals every internal ranking signal.²

Anthropic currently documents three tool versions: `web_search_20250305` for basic search, `web_search_20260209` for dynamic filtering, and `web_search_20260318` for response inclusion. Tool version, model, location, allowed or blocked domains, and the rest of the request belong in the evidence record because changing any of them changes the measurement surface.²

Publisher controls: choose the use, then test the delivery path

A common publisher policy is to allow search discovery and user-directed retrieval while declining model-training collection. That means independent records for Claude-SearchBot, Claude-User, and ClaudeBot, not one wildcard block. Anthropic supports the non-standard `Crawl-delay` directive, but a delay is a rate preference rather than evidence that every edge, proxy, or origin request will succeed.¹

Apply and test the rules per subdomain. A robots.txt file on `www.example.com` does not govern `docs.example.com`, and the page Anthropic needs may live behind a different CDN or WAF policy from the marketing homepage. Robots permission also cannot override authentication, a bot challenge, a 403, a redirect loop, an invalid canonical, or content that the response never exposes.¹

Do not use IP blocking as the opt-out mechanism. Anthropic warns that blocking its IPs can prevent its systems from reading robots.txt and therefore from learning the publisher's durable preference. Current IP ranges can still help verify that a request claiming an Anthropic user agent is genuine, but the robots policy should express the use decision and ordinary access controls should protect genuinely private content.¹

For removal from web-search outputs, Anthropic advises publishers to remove or password-protect the content, use `noindex` so search partners do not index it, block the applicable Anthropic bots, or request URL removal through its published process. `noindex` is broader than an Anthropic-only search preference and may still allow Claude to link to a URL or visit it directly, so choose it only when that wider indexing consequence matches the intent.³

# Allow discovery for Claude search
User-agent: Claude-SearchBot
Allow: /
Crawl-delay: 1

# Allow a page to be fetched when a user asks Claude for it
User-agent: Claude-User
Allow: /

# Decline future collection for model training
User-agent: ClaudeBot
Disallow: /

What makes an accessible page competitive

Once the intended crawler can reach the page, work at the claim level. Give a direct answer in visible text, then supply the material an answer system needs to use it responsibly: scope, definitions, criteria, dates, limitations, examples, methodology, and links to primary evidence. A short extractable sentence helps retrieval; the evidence around it helps selection and verification.

Keep the publisher, author or accountable organization, canonical URL, and meaningful update date clear. Align names, product facts, prices, and availability across the page and credible external profiles. Structured data can label visible facts consistently, but Anthropic's reviewed search and crawler documentation does not identify a special Claude citation schema. Markup supports machine-readable consistency; it does not buy selection.

Compare observed cited pages with yours for the particular question. Do they provide first-party data, a useful taxonomy, direct experience, a more current comparison, stronger attribution, or a clearer answer to the constraint? Close a supported evidence gap instead of copying their word count or publishing thin variations for every prompt. External corroboration may matter, but it can also become the source Claude cites instead of the publisher's own page.

• Give each important information need one stable, public canonical URL.
• Put the answer and its qualifications in server-readable visible text, not only an image or interaction.
• Name the accountable author or organization and date claims that can become stale.
• Support consequential claims with primary documentation, transparent first-party data, or a reproducible method.
• Use descriptive internal links from relevant hubs so readers and crawlers can discover the page in context.
• Keep visible facts, metadata, structured data, and reputable external profiles consistent.
• Review the sources Claude cites for evidence gaps, not templates to imitate.

Measurement: preserve errors and label the surface

Anthropic documents an important API edge case: a web-search tool error can appear inside an otherwise successful HTTP 200 response. A valid search with no results is represented by an empty result list, which is different from a tool error. If a response ends with `pause_turn`, the caller must return the assistant content unchanged so Claude can continue. A measurement pipeline that looks only at the HTTP status can therefore misclassify incomplete or failed retrieval as a clean absence.²

For every answer check, preserve the exact query, provider, surface, model, tool version, request settings, location, observation time, answer text, citation annotations, all returned source URLs, target-domain outcome, and provider status. Store timeout, rate-limit, tool-error, malformed, and paused outcomes as unavailable or incomplete evidence. Never turn a provider error into ‘not cited’.

Label an Anthropic Messages API check as an API surface, not as claude.ai. Label a captured consumer session with its account state, mode, location, conversation context, visible source links, and time. Results can be compared within a stable surface; they should not be silently pooled across API models, tool versions, consumer modes, or changing request settings.

Keep server logs and analytics parallel to answer observations. A verified Claude-SearchBot request is Eligible access evidence. A Claude-User fetch is user-directed retrieval evidence. An answer citation is Observed attribution. A referral is a visit. A conversion is a downstream outcome. These events can form a useful sequence, but the existence of one does not prove the others or establish causality.

First-party observation: Claude cited a competitor

On 10 August 2026 at 09:38 Europe/London, Robot Visible's own monitoring record stored a Claude answer to “How do I track changes to my website's AI visibility over time?” The provider returned a substantive answer about fixed prompt sets, mentions, citations, and longitudinal monitoring. Robot Visible was neither mentioned nor cited. The answer cited `https://otterly.ai/`, which the monitor recorded as a competitor source.

The result is useful precisely because it is bounded. It is one stored API answer from Anthropic Messages with web search, not a claude.ai consumer session, an index report, or a population-wide ranking. It shows an observed competitor citation for one query and time. It does not show which crawler discovered or fetched the source, whether a crawler rule caused the outcome, how often other users saw it, or whether the result persists.

The production monitor preserved the provider label, API surface, exact question, answer excerpt, cited URL, target-domain outcome, and run time. Its current adapter uses the `claude-sonnet-5` model and `web_search_20250305` tool with at most five searches; those are the settings in the reproduction below. A fresh run is a new dated observation, not a demand for identical prose or sources.

How to reproduce and extend the observation

The request below matches the monitored surface: Anthropic's Messages endpoint, the `claude-sonnet-5` model, the basic `web_search_20250305` tool, and a five-search cap. Save the whole response securely, including content blocks, citation annotations, stop reason, usage, request ID from response headers, HTTP status, and your own observation timestamp. The request is billable, and Anthropic's prices and model availability can change.²

When parsing the result, treat `web_search_tool_result` blocks containing `is_error` as provider errors even if the HTTP response was 200. Preserve an empty successful result list separately. If the stop reason is `pause_turn`, return the received assistant content unchanged in the next Messages request before judging the final answer. Only classify the target as absent after a usable final answer exists.²

Repeat the same request on a schedule and after material changes, storing every attempt. Compare citations by normalized URL and domain, but retain the original URLs and cited text. If you also inspect claude.ai, record that as another surface. Differences are evidence about variation, not failed reproductions.

const observedAt = new Date().toISOString();
const response = await fetch("https://api.anthropic.com/v1/messages", {
  method: "POST",
  headers: {
    "x-api-key": process.env.ANTHROPIC_API_KEY,
    "anthropic-version": "2023-06-01",
    "content-type": "application/json",
  },
  body: JSON.stringify({
    model: "claude-sonnet-5",
    max_tokens: 2048,
    messages: [{
      role: "user",
      content: "How do I track changes to my website's AI visibility over time?",
    }],
    tools: [{
      type: "web_search_20250305",
      name: "web_search",
      max_uses: 5,
    }],
  }),
});

const result = await response.json();
console.log(JSON.stringify({
  observedAt,
  httpStatus: response.status,
  requestId: response.headers.get("request-id"),
  model: result.model,
  stopReason: result.stop_reason,
  content: result.content,
  usage: result.usage,
}, null, 2));

The practical Claude search workflow

• Choose separate policies for Claude-SearchBot, Claude-User, and ClaudeBot based on search, user retrieval, and training intent.
• Publish and verify robots.txt on every relevant subdomain; use Crawl-delay only when it reflects an intentional rate policy.
• Test exact canonical URLs through DNS, CDN, WAF, redirects, authentication, rendering, and origin response layers.
• Use current Anthropic IP data to verify claimed requests, not as the durable opt-out mechanism.
• Keep direct answers, qualifications, authorship, dates, methodology, and primary evidence visible and internally discoverable.
• Compare observed cited sources claim by claim and close a real evidence gap rather than imitating page structure.
• Monitor a stable question set and label API and claude.ai consumer observations as separate surfaces.
• Preserve tool errors, empty results, pause_turn, mentions, citations, referrals, and conversions as distinct outcomes.
• After publishing a change, collect multiple comparable observations before describing persistence or attributing cause.

Run a free Claude crawler and page-readiness check