citelity.

Claude-SearchBot vs ClaudeBot vs Claude-User: what to allow

Anthropic runs three crawlers, and Claude's web search also draws on Brave. What each bot does, what blocking it costs according to Anthropic, and how to check Claude can see you.

By Ed Grows · Updated 8 min read

Anthropic runs three web crawlers, and only one of them is about training. ClaudeBot collects content that may be used to train models. Claude-SearchBot indexes pages to improve Claude's search results. Claude-User fetches a page live when someone asks Claude a question that needs it. If you want Claude to cite you, allow the second and third. Blocking the first keeps you out of future training data, and Anthropic's crawler documentation (last updated 7 April 2026) describes it separately from search.

TRAININGClaudeBotCollects content that maybe used for training.IF YOU BLOCK ITFuture content excludedfrom training datasets.Your call.SEARCH INDEXINGClaude-SearchBotIndexes pages to improveClaude's search results.IF YOU BLOCK ITNot indexed for search;visibility may drop.Allow it.USER-DIRECTED FETCHClaude-UserFetches a page live whena user's question needs it.IF YOU BLOCK ITClaude can't read yourpage to answer a user.Allow it.

What each Anthropic crawler does, in Anthropic's words

ClaudeBot. It "helps enhance the utility and safety of our generative AI models by collecting web content that could potentially contribute to their training." Restricting it "signals that the site's future materials should be excluded from our AI model training datasets."

Claude-SearchBot. It "navigates the web to improve search result quality for users." Blocking it "prevents our system from indexing your content for search optimization, which may reduce your site's visibility and accuracy in user search results."

Claude-User. "When individuals ask questions to Claude, it may access websites using a Claude-User agent." Blocking it "prevents our system from retrieving your content in response to a user query, which may reduce your site's visibility for user-directed web search."

Two details in that wording matter. First, Anthropic says "may reduce", not "removes": Claude-SearchBot is not described as the only route into Claude's answers (see the Brave section below). Second, Claude-User is not just for pasted links. It fetches pages to answer questions, so blocking it can cut you out of answers even when your page is indexed.

The same document says all three respect robots.txt and honor the non-standard Crawl-delay directive. Anthropic asks site owners not to block by IP address, because that also stops its crawlers from reading robots.txt. It publishes its crawler IP ranges at claude.com/crawling/bots.json for allowlisting.

The robots.txt that keeps you citable

For a site that wants out of training but into Claude's answers:

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

Claude-SearchBot and Claude-User were documented in 2025, so older "block AI crawlers" templates never named them. Those templates still catch them through a broad User-agent: * disallow, or through rules aimed at "Anthropic" in general. Read your file for both.

AI
Free tool · No signup
Free AI Crawler Checker
Paste a domain → see which AI crawlers your robots.txt allows and which it blocks, with training and retrieval bots told apart.
Check your crawlers →

Where Claude's search results come from

Partly from Brave Search. In March 2025 Anthropic added Brave Search to its list of subprocessors, and developers found a BraveSearchParams parameter in Claude's web search, plus at least one search where Claude and Brave returned identical citations (TechCrunch, 21 March 2025). Anthropic has not published how Brave results and its own Claude-SearchBot index are combined, so treat them as two inputs.

The Brave link has a consequence most Claude guides miss. Brave's crawler does not announce its own user agent, and Brave says that "if a domain or page is not crawlable by Googlebot, then Brave Search's bot will not crawl it either." A rule that blocks Googlebot from a section of your site therefore keeps that section out of Brave, and plausibly out of Claude's search results. Brave also says robots.txt is not how you keep a page out of its index; it uses the noindex directive for that.

A quick self-check: search site:yourdomain.com on search.brave.com and confirm your important pages appear.

How big is Claude as a traffic source?

Small, and growing fast. In SE Ranking's panel of 101,574 sites (published June 2026), Claude referral traffic grew 386% between January and April 2026, yet it still accounted for 1.40% of AI-referred traffic, against 78.23% for ChatGPT. Similarweb reported Claude's US monthly users up 349% between June 2025 and May 2026 (PPC Land).

Web search itself is not new: Anthropic launched it on 20 March 2025 for paid US users and made it available on all plans worldwide on 27 May 2025 (Claude blog). Developers building on the Claude API have to switch web search on as a paid tool, so not every product built on Claude searches the web. When one does show your page, the visit carries that product's referrer, not claude.ai's. Your analytics will undercount Claude for that reason, by an amount nobody can measure.

What is not known about Claude citations

Anthropic publishes no citation statistics. None of the large public studies of where on a page engines quote from, or how fresh the cited pages are, included Claude. One that did include it, SparkToro and Gumshoe's January 2026 test of about 3,000 runs across ChatGPT, Claude and Google's AI, found the same list of recommendations came back less than once in 100 runs (Search Engine Land). So a single Claude check tells you little. Measure a rate, as described in how to track AI citations.

Anything more specific you read, such as how often Claude searches or how strictly it filters sources, is practitioner impression. The page-level work that applies to every engine (direct answers, self-contained sections, attributed facts) is covered in what is AEO.

Your Claude checklist

  1. robots.txt: Claude-SearchBot and Claude-User allowed, including under any User-agent: * rule. ClaudeBot as you choose.
  2. Googlebot not blocked from anything you want Claude to cite, because Brave follows Googlebot's rules.
  3. No IP-level blocks on Anthropic ranges; CDN AI-bot settings checked.
  4. Server logs show Claude-SearchBot and Claude-User requests with 200 responses.
  5. Your key pages show up for site: searches on search.brave.com.
  6. Ten real questions run in claude.ai with web search on, repeated over a few days, with the cited domains recorded.
Can I slow Anthropic's crawlers down instead of blocking them?
Yes. Anthropic says its crawlers honor the non-standard Crawl-delay directive in robots.txt. That reduces load without removing you from Claude's search index or blocking live fetches for user questions.
Does blocking ClaudeBot remove content Anthropic has already collected?
Anthropic's documentation says a ClaudeBot block signals that the site's future materials should be excluded from training datasets. It does not describe removing content collected before the block. The block also does not affect Claude-SearchBot or Claude-User, which have their own robots.txt entries.

Sources

Updated October 1, 2026: rebuilt around Anthropic's own descriptions of ClaudeBot, Claude-SearchBot and Claude-User; corrected the claim that blocking Claude-User only affects pasted links; added Claude's Brave Search dependency and the Googlebot rule that comes with it.

Keep reading

aeo · 7 min readGoogle-Extended vs Googlebot: what actually controls Gemini citationsBlocking Google-Extended doesn't touch AI Overviews, but it can keep you out of the Gemini app's grounded answers. Google's three controls, the surfaces each one governs, and how to test Gemini.aeo · 10 min readAEO content checklist: 47 checks, from crawler access to quarterly upkeep47 AEO checks in the order you do them: retrievability (now including Search Console's AI control), planning, structure, schema, authorship, validation, upkeep.aeo · 14 min readAEO for affiliate sites: the citation and recovery playbookWhat gets an affiliate review cited in AI answers, what the 2026 spam and review rules changed, and a corrected checklist for rebuilding a site that lost traffic.