Guide · Practitioner
How to Rank in Claude
To be cited by Claude, allow Anthropic's search and user-request bots (Claude-SearchBot and Claude-User) in robots.txt, then publish clear pages with current facts. Anthropic says Claude searches the web when a question benefits from current information and includes citations in every search response (Anthropic). Blocking ClaudeBot only affects training (Anthropic).
Facts are from Anthropic's help centre, accessed 30 Oct 2026. Recommendations marked "our view" are interpretation.
Which Anthropic bots matter?
Anthropic documents three robots (Anthropic):
| Bot | Purpose | Effect of blocking (per Anthropic) |
|---|---|---|
| ClaudeBot | Collects content that may contribute to model training | Future materials excluded from training datasets |
| Claude-User | Fetches pages when users ask Claude questions | Prevents retrieval for user queries; may reduce visibility in user-directed search |
| Claude-SearchBot | Indexes content to improve search result quality | Prevents indexing for search; may reduce visibility and accuracy in search results |
So you can opt out of training (block ClaudeBot) while staying visible in search (allow the other two).
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /How should you configure access?
Anthropic says its bots honour robots.txt and support the non-standard Crawl-delay (Anthropic). Key points from its guidance:
- Add rules for every subdomain you want to control.
- Use robots.txt, not IP blocking. Anthropic warns IP blocks may not work reliably because they stop it reading your robots.txt.
- Anthropic says its bots respect anti-circumvention tools such as CAPTCHAs. Our view: that means a CAPTCHA or bot challenge in front of your content will keep Claude out.
When does Claude search?
Anthropic says (Anthropic):
| Situation | Behaviour |
|---|---|
| Questions that benefit from current information | Claude invokes a search tool to ground the response |
| New Claude experience | No toggle; Claude searches when it helps |
| Team and Enterprise plans | An Owner must enable web search for the workspace |
| User says "search the web" | Claude uses web search |
| User provides a URL | Web fetch can retrieve the full page |
| Localised requests | Claude may use location inferred from IP |
What do Claude's search answers look like?
Anthropic says Claude processes multiple sources and replies with direct citations, source links and relevant quotes where appropriate (Anthropic). Image results are powered by Bing and include source links.
Our view: because responses can include quotes, pages with crisp, quotable sentences (definitions, facts, numbers with dates) are easier to cite.
What content does Claude cite well?
Our view, based on the answer format above:
- [ ] Direct answer in the first paragraph
- [ ] Short, self-contained statements that make sense when quoted
- [ ] Dates on facts and pages
- [ ] Tables for comparisons and specs
- [ ] Authorship and evidence for claims
- [ ] No content hidden behind login walls or challenges
How do you test Claude visibility?
- Turn on web search (or use the new experience) and add "search the web" to prompts for consistency.
- Run 20–50 buyer prompts (prompt library).
- Record citations, quoted text and whether your domain appears.
- Check logs for Claude-SearchBot and Claude-User hits and status codes.
- Re-run monthly.
| Prompt | Searched? | Your domain cited? | Quoted text |
|---|---|---|---|
| "best X for Y" | Yes | No | — |
Illustrative row.
What's a worked plan?
Illustrative example: a cybersecurity vendor blocked "all AI bots" with a single robots.txt rule list copied from a blog post. Claude never cites it.
- Splits the rules: blocks ClaudeBot (training), allows Claude-SearchBot and Claude-User.
- Removes a bot-challenge rule on its docs subdomain and adds matching robots.txt there.
- Rewrites its top three product pages with quotable one-sentence answers and dated specs.
- Re-tests after a month.
What doesn't help?
- Blocking Anthropic IPs instead of using robots.txt
- Blocking all three bots and expecting citations
- Long, unstructured pages without clear statements
How do you check whether Claude can reach you?
Work through this in order:
- robots.txt: confirm no rule for
*or a list of AI agents disallows Claude-SearchBot or Claude-User. Check each subdomain's own file, as Anthropic asks for rules per subdomain (Anthropic). - Server logs: search for the Claude-SearchBot and Claude-User user agents. Note status codes; repeated 403 or 429 responses suggest a firewall or rate limit is blocking them.
- Bot challenges: if your CDN shows a challenge page to unknown bots, Anthropic's bots will not bypass it (Anthropic).
- Fetch test: paste a URL into Claude with web search on and ask for a summary. If Claude can't read it, fix access before working on content.
Our view: most "Claude never cites us" problems we see start with access, not content quality. Fix access first, because content work can't help a page Claude can't read.
Checklist
- [ ] Bot policy per Anthropic agent
- [ ] Rules on every subdomain
- [ ] No CAPTCHA in front of public content
- [ ] Quotable, dated content
- [ ] Monthly tests and log checks
Related
Sources
- Anthropic, Does Anthropic crawl data from the web? (7 Apr 2026); Enable and use web search
All accessed 30 Oct 2026.
Questions
How do I rank in Claude?
Allow Claude-SearchBot and Claude-User in robots.txt, publish clear pages that answer specific questions with current facts, and earn coverage on authoritative sites Claude is likely to cite.
Does blocking ClaudeBot remove me from Claude's answers?
ClaudeBot is for training data. Anthropic says blocking it excludes future content from training. Search visibility depends on Claude-SearchBot and Claude-User.
Does Claude always search the web?
No. Claude searches when a question benefits from current information, or when web search is on and the user asks it to.