ClaudeBot and Claude-SearchBot are two of the three bots that Anthropic, the company behind Claude, describes in its help article on crawling. ClaudeBot is for training, Claude-SearchBot for search. Each bot has its own purpose, so blocking one has a different effect from blocking another:
- ClaudeBot collects web content that may contribute to training. Blocking it signals that your future content should be excluded from training data.
- Claude-SearchBot crawls the web to improve search results for users. Blocking it prevents your content from being indexed for that purpose, which can reduce your visibility in search answers.
- Claude-User fetches pages when a user asks Claude a question. Blocking it prevents Claude from retrieving your content for such a question.
Anthropic writes that its bots follow standard robots.txt rules and also support the non-standard Crawl-delay rule. According to Anthropic, blocking by IP address does not work reliably; robots.txt is the recommended method.
Example
Refusing training, allowing search:
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /Common mistake
Blocking only ClaudeBot and assuming Claude will no longer use your site, or blocking all three and expecting Claude to keep linking to your pages. Decide per bot what you want.
More in Allowing or blocking AI crawlers with robots.txt and under AI crawler.