Glossary · Beginner
What Is ClaudeBot? Definition and Example
ClaudeBot is the crawler Anthropic uses to collect web content that could contribute to training its generative AI models. Anthropic says restricting ClaudeBot signals that a site's future materials should be excluded from training datasets (Anthropic). It's one of three Anthropic agents, each with a separate purpose.
Anthropic's three agents
| Agent | Purpose (per Anthropic) | Effect of blocking |
|---|---|---|
| ClaudeBot | Collect content that may be used for model training | Future content excluded from training |
| Claude-SearchBot | Improve search result quality for users | May reduce visibility and accuracy in search results |
| Claude-User | Fetch pages when a user asks Claude a question | Claude can't retrieve your content for those queries |
Example robots.txt
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /Illustrative: a site opting out of training while staying available to Claude's search and user requests. Anthropic notes you need to apply rules for every subdomain you want to opt out.
How does Anthropic say its bots behave?
Anthropic states its bots honour robots.txt, respect anti-circumvention technologies such as CAPTCHAs, and support Crawl-delay to limit crawl rate. It warns that blocking its IP addresses may not reliably opt you out, because it stops the bots reading your robots.txt.
Why it matters for GEO
Blocking "ClaudeBot" because it's the best-known name is a decision about training, not about whether Claude can find you. If Claude visibility matters, check the rules for Claude-SearchBot and Claude-User separately.
How do you check Anthropic's bots are reaching you?
Search your access logs for ClaudeBot, Claude-SearchBot and Claude-User, and check the response codes. A string of 403 responses usually means a firewall or CDN rule is blocking them, regardless of what robots.txt says. Anthropic says crawlers with a source IP on its published list are coming from Anthropic, which helps rule out impersonators.
Related
Questions
Does blocking ClaudeBot stop Claude citing my site?
Not necessarily. Anthropic uses separate agents for search (Claude-SearchBot) and user-initiated fetches (Claude-User).
Does Anthropic support Crawl-delay?
Yes. Anthropic says it supports the non-standard Crawl-delay extension.
Will blocking Anthropic's IP addresses opt me out?
Anthropic says IP blocking may not work reliably, because it prevents its bots reading robots.txt. Use robots.txt.