# Claude-SearchBot

## Quick answer

Claude-SearchBot is Anthropic's crawler for improving the quality of Claude's search results. It is separate from ClaudeBot, which gathers training data, and from Claude-User, which fetches a page a person or Claude itself opened directly during a conversation, and each follows its own robots.txt rule.

## What it does

Claude-SearchBot navigates the web specifically to improve the quality of the search results Claude returns when a person asks it a question that benefits from current web information. That makes it distinct from ClaudeBot, which Anthropic uses to gather potential training data, and from Claude-User, which fetches a single page because a user or Claude's own browsing asked for it directly. Claude-SearchBot respects standard robots.txt directives, so a site can control whether this specific bot is allowed to crawl it independently of the other two. Anthropic publishes a combined bots.json file listing its crawlers' IP ranges, so a site owner can confirm that traffic claiming to be Claude-SearchBot is genuine rather than a spoofed user agent. MarketHQ tracks how brands appear in Claude's answers as part of its wider AI visibility tracking across ChatGPT, Claude, Gemini, Perplexity, Grok, and DeepSeek.

## Facts

- Operator: Anthropic
- Robots token: Claude-SearchBot
- User agent string(s): Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +Claude-SearchBot@anthropic.com)
- Purpose: search-index
- Respects robots.txt: yes
- IP ranges: https://claude.com/crawling/bots.json

## Should you block it?

Disallowing Claude-SearchBot can reduce how accurately and how often a site shows up when someone asks Claude a question that triggers a web search, since Anthropic uses this crawler specifically to keep those search results current. For a brand that wants to be found when people ask Claude about its category or compare it to competitors, allowing Claude-SearchBot keeps that door open; disallowing it risks Claude answering from older or incomplete information about the brand. The tradeoff for a site is usually not about AI training, since Claude-SearchBot is separate from ClaudeBot and Anthropic documents it as a search-quality crawler rather than a training one. A site mainly worried about training data reuse can disallow ClaudeBot specifically while still allowing Claude-SearchBot, keeping search visibility without opening the door to training use.

## How to block it

Add this to robots.txt to block Claude-SearchBot:

User-agent: Claude-SearchBot
Disallow: /

This rule only affects Claude-SearchBot; ClaudeBot and Claude-User are controlled by separate rules naming those tokens.

## FAQ

### Is Claude-SearchBot the same as ClaudeBot?

No. Claude-SearchBot improves Claude's search result quality. ClaudeBot gathers potential training data. Each is controlled by its own robots.txt rule.

### Can I verify traffic claiming to be Claude-SearchBot?

Yes. Anthropic publishes its crawlers' IP ranges at claude.com/crawling/bots.json.

### Does blocking Claude-SearchBot affect AI training?

No. Anthropic documents it as a search-quality crawler, separate from ClaudeBot, which is used for training data.

## Sources

- Claude-SearchBot crawls the web to improve the quality of Claude's search results. Source: https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler