Claude-SearchBot
Anthropic's search / answer indexing crawler.
Quick answer
Claude-SearchBot is Anthropic's crawler for improving the quality of Claude's search results. It is separate from ClaudeBot, which gathers training data, and from Claude-User, which fetches a page a person or Claude itself opened directly during a conversation, and each follows its own robots.txt rule.
What it does
Claude-SearchBot navigates the web specifically to improve the quality of the search results Claude returns when a person asks it a question that benefits from current web information. That makes it distinct from ClaudeBot, which Anthropic uses to gather potential training data, and from Claude-User, which fetches a single page because a user or Claude's own browsing asked for it directly. Claude-SearchBot respects standard robots.txt directives, so a site can control whether this specific bot is allowed to crawl it independently of the other two. Anthropic publishes a combined bots.json file listing its crawlers' IP ranges, so a site owner can confirm that traffic claiming to be Claude-SearchBot is genuine rather than a spoofed user agent. MarketHQ tracks how brands appear in Claude's answers as part of its wider AI visibility tracking across ChatGPT, Claude, Gemini, Perplexity, Grok, and DeepSeek.
Facts
Operator
Anthropic
Robots token
Claude-SearchBot
User agent string(s)
- Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +Claude-SearchBot@anthropic.com)
Purpose
Search / answer indexing
Respects robots.txt
Yes
IP ranges
https://claude.com/crawling/bots.json
From Anthropic's documentation, checked Oct 2026.
Should you block it?
Disallowing Claude-SearchBot can reduce how accurately and how often a site shows up when someone asks Claude a question that triggers a web search, since Anthropic uses this crawler specifically to keep those search results current. For a brand that wants to be found when people ask Claude about its category or compare it to competitors, allowing Claude-SearchBot keeps that door open; disallowing it risks Claude answering from older or incomplete information about the brand. The tradeoff for a site is usually not about AI training, since Claude-SearchBot is separate from ClaudeBot and Anthropic documents it as a search-quality crawler rather than a training one. A site mainly worried about training data reuse can disallow ClaudeBot specifically while still allowing Claude-SearchBot, keeping search visibility without opening the door to training use.
How to block it
Add this to robots.txt to block Claude-SearchBot: User-agent: Claude-SearchBot Disallow: / This rule only affects Claude-SearchBot; ClaudeBot and Claude-User are controlled by separate rules naming those tokens.
Frequently asked questions
Is Claude-SearchBot the same as ClaudeBot?
No. Claude-SearchBot improves Claude's search result quality. ClaudeBot gathers potential training data. Each is controlled by its own robots.txt rule.
Can I verify traffic claiming to be Claude-SearchBot?
Yes. Anthropic publishes its crawlers' IP ranges at claude.com/crawling/bots.json.
Does blocking Claude-SearchBot affect AI training?
No. Anthropic documents it as a search-quality crawler, separate from ClaudeBot, which is used for training data.
Related
- AI crawler directory — every AI crawler's robots.txt token, purpose and facts in one place.
- Free AI crawler checker — check which AI crawlers a domain's robots.txt actually allows or blocks.
- Glossary — definitions of the AI-visibility terms that come up alongside crawler behavior.
Sources
- Claude-SearchBot crawls the web to improve the quality of Claude's search results. Source: https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler (checked Oct 2026)
MarketHQ tracks brand mentions across communities, news, blogs, social and AI answers, and turns them into gap analysis and action plans.