# How to Rank in Claude

> How Claude's web search finds and cites pages, the three Anthropic crawlers and what blocking each does, and a practical plan to be cited by Claude.

Source: https://mentionx.ai/learn/engines/claude/how-to-rank  
Author: Dipuck Jones, founder of MentionX  
Updated: 17 November 2026  
Level: Practitioner

**To be cited by Claude, allow Anthropic's search and user request bots (Claude-SearchBot and Claude-User) in robots.txt, then publish clear pages with current facts.** Anthropic says Claude searches the web when a question benefits from current information and includes citations in every search response ([Anthropic](https://support.claude.com/en/articles/10684626-enable-and-use-web-search)). Blocking ClaudeBot only affects training ([Anthropic](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)).

> **Key takeaways**
>
> - Claude can only cite pages its search and user request bots, Claude-SearchBot and Claude-User, are allowed to fetch.
> - Anthropic says its bots honour robots.txt and won't bypass bot challenges, so check robots.txt and your CDN first.
> - MentionX tracks Claude as an addon engine. The dataset behind this guide holds no Claude answers, so it uses figures from six other engines to show the patterns that apply to any search backed assistant.
> - Across those engines, blog posts and articles were the most cited page type (19.6% of 9,750 citations), and comparison prompts named the tracked brand in 91.9% of answers.
> - Test with a fixed prompt set and repeated runs, and log the cited URLs and quoted text.

*Checked on 31 October 2026. "Our view" marks MentionX opinion. Ledgerline is a fictional company used for illustration.*

## Which Anthropic bots matter?

Anthropic documents three robots ([Anthropic](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)):

| Bot | Purpose | Effect of blocking (per Anthropic) |
|---|---|---|
| ClaudeBot | Collects content that may contribute to model training | Future materials excluded from training datasets |
| Claude-User | Fetches pages when users ask Claude questions | Prevents retrieval for user queries; may reduce visibility in user directed search |
| Claude-SearchBot | Indexes content to improve search result quality | Prevents indexing for search; may reduce visibility and accuracy in search results |

So you can opt out of training (block ClaudeBot) while staying visible in search (allow the other two).

```
User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /
```

## How should you configure access?

Anthropic says its bots honour robots.txt and support the nonstandard Crawl-delay ([Anthropic](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)). Key points from its guidance:

- Add rules for **every subdomain** you want to control.
- Use robots.txt, not IP blocking. Anthropic warns IP blocks may not work reliably because they stop it reading your robots.txt.
- Anthropic says its bots respect anticircumvention tools such as CAPTCHAs. *Our view:* that means a CAPTCHA or bot challenge in front of your content will keep Claude out.

Since 30 October 2026 Anthropic also recognises a shared "no AI training" signal in robots.txt. It concerns training only and leaves Claude-SearchBot and Claude-User to their own rules; see [the AI training opt out in robots.txt](https://mentionx.ai/learn/technical/ai-training-opt-out-robots-txt-signal).

## When does Claude search?

Anthropic says ([Anthropic](https://support.claude.com/en/articles/10684626-enable-and-use-web-search)):

| Situation | Behaviour |
|---|---|
| Questions that benefit from current information | Claude invokes a search tool to ground the response |
| New Claude experience | No toggle; Claude searches when it helps |
| Team and Enterprise plans | An Owner must enable web search for the workspace |
| User says "search the web" | Claude uses web search |
| User provides a URL | Web fetch can retrieve the full page |
| Localised requests | Claude may use location inferred from IP |

## What do Claude's search answers look like?

Anthropic says Claude processes multiple sources and replies with direct citations, source links and relevant quotes where appropriate ([Anthropic](https://support.claude.com/en/articles/10684626-enable-and-use-web-search)). Image results are powered by Bing and include source links.

*Our view:* because responses can include quotes, pages with crisp, quotable sentences (definitions, facts, numbers with dates) are easier to cite.

## What content does Claude cite well?

*Our view, based on the answer format above:*

- Direct answer in the first paragraph
- Short, self contained statements that make sense when quoted
- Dates on facts and pages
- Tables for comparisons and specs
- Authorship and evidence for claims
- No content hidden behind login walls or challenges

## How do you test Claude visibility?

1. Turn on web search (or use the new experience) and add "search the web" to prompts for consistency.
2. Run 20 to 50 buyer prompts ([prompt library](https://mentionx.ai/learn/engines/chatgpt/buyer-prompt-library-50-prompts-test)).
3. Record citations, quoted text and whether your domain appears.
4. Check logs for Claude-SearchBot and Claude-User hits and status codes.
5. Rerun monthly.

| Prompt | Searched? | Your domain cited? | Quoted text |
|---|---|---|---|
| "best AP automation for midsized companies" | Yes | No | n/a |

*Ledgerline example; Ledgerline is fictional.*

## What doesn't help?

- Blocking Anthropic IPs instead of using robots.txt
- Blocking all three bots and expecting citations
- Long, unstructured pages without clear statements

## How do you check whether Claude can reach you?

Work through this in order:

1. **robots.txt:** confirm no rule for `*` or a list of AI agents disallows Claude-SearchBot or Claude-User. Check each subdomain's own file, as Anthropic asks for rules per subdomain ([Anthropic](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)).
2. **Server logs:** search for the Claude-SearchBot and Claude-User user agents. Note status codes; repeated 403 or 429 responses suggest a firewall or rate limit is blocking them.
3. **Bot challenges:** if your CDN shows a challenge page to unknown bots, Anthropic's bots will not bypass it ([Anthropic](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)).
4. **Fetch test:** paste a URL into Claude with web search on and ask for a summary. If Claude can't read it, fix access before working on content.

*Our view:* most "Claude never cites us" problems we see start with access, not content quality. Fix access first, because content work can't help a page Claude can't read.

## From MentionX data: what search backed assistants cite

**Method.** Read only aggregate queries over MentionX scans from 22 August to 29 October 2026 (UTC): 2,217 engine runs (2,154 completed) across 18 anonymised projects and six engines.

This dataset holds no Claude answers, so there are no Claude specific rates here. What it shows is how six other engines behaved, which gives a baseline for what to expect from any assistant that searches the web.

| Engine | Brand mentioned | Answers with no citations |
| --- | --- | --- |
| Gemini | 47.3% | 50.7% |
| OpenAI | 42.9% | 30.2% |
| Perplexity | 31.5% | 4.1% |
| AI Mode | 31.3% | 12.6% |
| Copilot | 26.0% | 47.0% |
| DeepSeek | 24.0% | 99.0% |

Brand mention rates ranged from 24.0% (DeepSeek) to 47.3% (Gemini). The share of answers without citations ranged even more widely, from 4.1% on Perplexity to 99.0% on DeepSeek. Across all engines, the page types cited most were:

| Cited URL type | Share of 9,750 citations |
| --- | --- |
| Blog or article | 19.6% |
| Homepage | 10.0% |
| Review page | 2.7% |
| Docs | 2.6% |
| Pricing | 1.9% |
| Wikipedia | 1.6% |

*Our view:* these differences are why you can't assume one engine's results carry over to another. Test Claude separately, with its own prompt set and log, rather than inferring from ChatGPT or Perplexity.

**Limits.** A convenience sample of mostly B2B software projects, run through engine APIs, which can behave differently from the consumer apps. Perplexity, Copilot and DeepSeek figures come from a single day (26 October 2026), so treat them as a snapshot, not a trend.

## Worked example: Ledgerline checks Claude access

*Ledgerline is a fictional accounts payable tool. Figures are illustrative.*

Ledgerline never appears as a source when Claude searches "best AP automation for midsized companies".

Week 1: the team finds a robots.txt block list copied from a blog post. It disallows ClaudeBot, Claude-SearchBot and Claude-User together. They decide to keep blocking ClaudeBot, which relates to training, and allow the other two.

Week 2: server logs show Claude-SearchBot now receiving 200 responses, except on the docs subdomain, which has its own robots.txt with the same old rules. They fix that file too.

Weeks 3 to 8: they rewrite the AP automation guide to open with a direct answer and a dated feature table, then run 20 prompts three times every two weeks, logging cited URLs and the quoted text.

> **Pro move:** Check robots.txt on every subdomain. Anthropic asks for rules per subdomain, and docs or help centres often carry their own outdated file.

## Why separate training from search?

Anthropic documents ClaudeBot, which collects content that may be used for training, separately from Claude-SearchBot and Claude-User, which fetch pages for search and user requests. That lets you make two separate decisions. Blocking training is a legitimate choice. Blocking search access means Claude can't fetch your pages when a user's question needs current information, so it can't cite them.

## Which pages should you check first?

Start with the pages most likely to answer buyer questions: product overviews, comparison pages, pricing, integration pages and your most useful guides. Across the engines we track, blog posts and articles were cited most (19.6% of citations), followed by homepages (10.0%); pricing pages made up 1.9% and docs 2.6%. Make sure the guides that get cited also state your key product facts in plain sentences, so a quote from them can support a recommendation.

## How do prompt types affect results?

Across the engines we track, the tracked brand was named far more often for some intents than others:

| Intent | Brand mentioned | Share of voice |
| --- | --- | --- |
| Brand specific | 100% | 64.3 |
| Comparison | 91.9% | 44.6 |
| Recommendation | 29.2% | 26.8 |
| Best of | 26.6% | 22.7 |
| Problem | 5.7% | 7.7 |

When testing Claude, include each intent. Brand specific prompts test accuracy. Comparison prompts against Competitor A to D test positioning. Unbranded best of and problem led prompts show whether you're considered at all, which is where the real gap usually is.

## How should you report Claude results?

Report the share of answers that searched the web, the share citing your domain, and the share naming your brand, each with the number of answers. Add the quoted text when Claude quotes you, because it shows which sentences are doing the work. Answers vary: across the engines we track, results changed between runs for 11 of 28 prompt engine pairs we retested, so one run per prompt isn't a result.

## How long until changes show?

Access fixes show once Claude's bots can fetch the page, which you can confirm in logs. Content changes depend on when the page is next fetched for a relevant question. We have no Claude timing data, so plan to retest every two weeks for at least two months before drawing conclusions.

## What makes a page easy for Claude to quote?

Anthropic says Claude replies with direct citations, source links and relevant quotes where appropriate. A quote needs a sentence worth quoting. That means:

- an answer in the first lines under each heading, written so it makes sense on its own;
- specific, dated facts rather than general claims;
- product names and categories stated plainly, so the quote carries your name with it;
- no key facts locked in images, PDFs or tabs that need scripts to open.

*Our view:* the quote test is a useful editing tool. Read each section's opening sentence alone and ask whether it would make sense in someone else's answer. If it relies on the paragraph before it, rewrite it.

## Who should own Claude work?

SEO or web operations owns robots.txt, CDN rules and log checks across every subdomain. Content owns the pages that answer buyer questions. Product marketing owns the facts. One person runs the test log, using the same prompts, settings and cadence each cycle, so results can be compared over time.

## Common mistakes

- Copying a "block all AI bots" list without checking what each bot does.
- Fixing the main robots.txt but not those on subdomains.
- Testing with search switched off, then concluding Claude ignores your site.
- Reporting one run per prompt as a result.

## How does Claude fit in a multi engine programme?

Treat it as one engine in a shared process. The access checks, content standards and test method are the same as for ChatGPT, Perplexity and Gemini; only the bots and the settings differ. Keep one prompt set across engines so you can compare like with like, and one log format so results sit side by side. Then decide where to spend effort based on where your buyers actually ask questions, which a "How did you hear about us?" field that lists AI assistants can help you find out.

## Common questions

**Does blocking ClaudeBot stop Claude citing me?** Anthropic documents ClaudeBot separately from Claude-SearchBot and Claude-User. Blocking only ClaudeBot shouldn't stop search access, but check your rules for each bot.

**Does Claude always search the web?** No. Anthropic describes when Claude searches; many answers come from the model without searching.

**Can I see Claude traffic in analytics?** Visits from cited links may show claude.ai as the referrer; add it to your AI assistants channel.

**Should I use Crawl-delay for Claude's bots?** Anthropic says its bots support it. Use it only if crawling causes load problems, since slowing access can delay how quickly new pages are fetched.

## Checklist

- Bot policy per Anthropic agent
- Rules on every subdomain
- No CAPTCHA in front of public content
- Quotable, dated content
- Monthly tests and log checks

## Related

- [How Claude picks sources and citations](https://mentionx.ai/learn/engines/claude/picks-sources-citations)
- [Why AI engines ignore your brand](https://mentionx.ai/learn/engines/chatgpt/ai-engines-ignore-brand-7-diagnoses)

## Sources

- Anthropic, [Does Anthropic crawl data from the web?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) (7 Apr 2026); [Enable and use web search](https://support.claude.com/en/articles/10684626-enable-and-use-web-search)
All accessed 30 Oct 2026.

## Frequently asked questions

### How do I rank in Claude?

Allow Claude-SearchBot and Claude-User in robots.txt, publish clear pages that answer specific questions with current facts, and earn coverage on authoritative sites Claude is likely to cite.

### Does blocking ClaudeBot remove me from Claude's answers?

ClaudeBot is for training data. Anthropic says blocking it excludes future content from training. Search visibility depends on Claude-SearchBot and Claude-User.

### Does Claude always search the web?

No. Claude searches when a question benefits from current information, or when web search is on and the user asks it to.

---

Written and reviewed by Dipuck Jones for MentionX Academy. The HTML version of this page: https://mentionx.ai/learn/engines/claude/how-to-rank
