Guide · Practitioner
ClaudeBot and Claude-SearchBot
Anthropic uses three agents: ClaudeBot collects web content that may be used to train its models, Claude-SearchBot crawls to improve search result quality for Claude users, and Claude-User fetches pages when a person asks Claude a question. Each is controlled separately in robots.txt (Anthropic). If you want Claude to find and cite your pages, Claude-SearchBot and Claude-User matter more than ClaudeBot.
Key takeaways
- ClaudeBot = training; Claude-SearchBot = search; Claude-User = user requests.
- Anthropic says blocking Claude-SearchBot or Claude-User may reduce your visibility in Claude's answers.
- Rules must be set on every subdomain you want covered.
- Anthropic supports
Crawl-delayand says its bots respect robots.txt and won't bypass CAPTCHAs.
What does each agent do?
| Agent | Anthropic's description | What blocking does |
|---|---|---|
| ClaudeBot | Collects web content that could potentially contribute to training | Signals future materials should be excluded from training datasets |
| Claude-SearchBot | Navigates the web to improve search result quality | Prevents indexing for search optimisation; may reduce visibility and accuracy in search results |
| Claude-User | Accesses websites when individuals ask Claude questions | Prevents retrieval in response to user queries; may reduce visibility for user-directed search |
Source: Anthropic Help Center.
Which robots.txt pattern fits your goals?
Visible in Claude, no training:
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /Everything allowed, with a gentler crawl rate:
User-agent: ClaudeBot
Crawl-delay: 1
Allow: /Opt out of all Anthropic agents:
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: Claude-User
Disallow: /Crawl-delay isn't part of the RFC 9309 standard (IETF) and Google ignores it, but Anthropic says it supports it.
Why do subdomains matter?
robots.txt applies per host. Anthropic explicitly says to add rules "for every subdomain that you wish to opt out from". A rule on www.example.com doesn't cover blog.example.com or docs.example.com. Illustrative: a SaaS company blocks ClaudeBot on its main site but forgets its docs subdomain, which holds most of its technical content. Its policy and its practice no longer match.
Why not block by IP?
Anthropic warns that alternative methods such as blocking its IP addresses "may not work correctly or persistently guarantee an opt-out", because blocking stops its bots reading robots.txt. Use robots.txt for preferences and keep IP information for verification: Anthropic says a crawler with a source IP on its published list is coming from Anthropic.
How do you check Anthropic traffic in your logs?
- Filter requests whose user agent contains
ClaudeBot,Claude-SearchBotorClaude-User. - Group by agent and response code.
- Check a sample of IPs against Anthropic's published list.
- Look for 403s or challenges from your CDN that contradict your robots.txt intent.
If you think a bot is misbehaving, Anthropic asks site owners to contact claudebot@anthropic.com from an email address on the affected domain.
How should you handle staging, private and duplicate hosts?
Because rules work per host, review every hostname that serves content publicly:
| Host type | Recommended approach |
|---|---|
| Staging or preview environments | Require authentication; don't rely on robots.txt |
| Docs or help-centre subdomains | Apply the same Anthropic policy as your main site, unless you've decided otherwise |
| Regional domains | Copy the groups to each domain's robots.txt |
| Legacy hosts that redirect | Make sure redirects work; robots.txt on the old host still applies to the old host |
Anthropic says its bots respect robots.txt and won't attempt to bypass CAPTCHAs, but authentication is the only real protection for content that must stay private. A public staging site with Disallow: / still exposes its URLs to anyone who reads the file.
How do you decide?
| Question | If yes, consider |
|---|---|
| Do you want Claude to cite your pages? | Allow Claude-SearchBot and Claude-User |
| Is your content licensed or your core product? | Discuss blocking ClaudeBot with legal |
| Is crawl load a concern? | Use Crawl-delay before blocking |
| Do you block other training crawlers? | Align ClaudeBot with that policy |
| Do you have many subdomains? | Keep one shared robots.txt snippet and deploy it to each host |
Whatever you choose, write it down with a date and an owner, and review it when Anthropic updates its article.
Common mistakes
- Blocking only ClaudeBot and assuming Claude can no longer see the site, or blocking all three when only training was the concern.
- Forgetting subdomains.
- Using IP blocks as an opt-out.
- Not reviewing the article; Anthropic updates it and offers a form to be notified of substantial changes.
Worked example
Illustrative. A training company sells online courses and doesn't want course text used for model training, but wants Claude to recommend its courses. It blocks ClaudeBot site-wide, allows Claude-SearchBot and Claude-User on its marketing pages and course catalogue, and disallows them on /lessons/, which sits behind a login anyway. It applies the same rules on its learn. subdomain and records the decision in a robots.txt comment.
Checklist
- [ ] Policy for training, search and user fetch decided
- [ ] Named groups for each Anthropic agent you treat differently
- [ ] Rules applied on every subdomain
- [ ] Crawl-delay considered before blocking for load
- [ ] Logs checked for response codes and verified IPs
Where to go next
Sources
Questions
Which Anthropic agent affects whether Claude can cite me?
Claude-SearchBot (search indexing) and Claude-User (user-initiated fetches). ClaudeBot relates to training.
Does Anthropic honour Crawl-delay?
Yes. Anthropic says it supports the non-standard Crawl-delay extension.
Should I block Anthropic by IP?
Anthropic says IP blocking may not reliably opt you out, because it stops its bots reading robots.txt.