GuidePractitioner11 min readPart of GEO Practitioner

How to Rank in Claude

How Claude's web search finds and cites pages, the three Anthropic crawlers and what blocking each does, and a practical plan to be cited by Claude.

Dipuck Jones
Written by Founder of MentionX · 13+ years in B2B SaaS growth and SEO
ReviewedUpdated
Short answer

To be cited by Claude, allow Anthropic's search and user-request bots (Claude-SearchBot and Claude-User) in robots.txt, then publish clear pages with current facts. Anthropic says Claude searches the web when a question benefits from current information and includes citations in every search response (Anthropic). Blocking ClaudeBot only affects training (Anthropic).

Key takeaways
  • Claude can only cite pages its search and user-request bots, Claude-SearchBot and Claude-User, are allowed to fetch.
  • Anthropic says its bots honour robots.txt and won't bypass bot challenges, so check robots.txt and your CDN first.
  • MentionX doesn't track Claude, so this guide uses figures from the six engines we do track to show the patterns that apply to any search-backed assistant.
  • Across those engines, blog posts and articles were the most cited page type (19.6% of 9,750 citations), and comparison prompts named the tracked brand in 91.9% of answers.
  • Test with a fixed prompt set and repeated runs, and log the cited URLs and quoted text.

Checked on 31 October 2026. "Our view" marks MentionX opinion. Ledgerline is a fictional company used for illustration.

01Which Anthropic bots matter?

Anthropic documents three robots (Anthropic):

BotPurposeEffect of blocking (per Anthropic)
ClaudeBotCollects content that may contribute to model trainingFuture materials excluded from training datasets
Claude-UserFetches pages when users ask Claude questionsPrevents retrieval for user queries; may reduce visibility in user-directed search
Claude-SearchBotIndexes content to improve search result qualityPrevents indexing for search; may reduce visibility and accuracy in search results

So you can opt out of training (block ClaudeBot) while staying visible in search (allow the other two).

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

02How should you configure access?

Anthropic says its bots honour robots.txt and support the non-standard Crawl-delay (Anthropic). Key points from its guidance:

  • Add rules for every subdomain you want to control.
  • Use robots.txt, not IP blocking. Anthropic warns IP blocks may not work reliably because they stop it reading your robots.txt.
  • Anthropic says its bots respect anti-circumvention tools such as CAPTCHAs. Our view: that means a CAPTCHA or bot challenge in front of your content will keep Claude out.

Anthropic says (Anthropic):

SituationBehaviour
Questions that benefit from current informationClaude invokes a search tool to ground the response
New Claude experienceNo toggle; Claude searches when it helps
Team and Enterprise plansAn Owner must enable web search for the workspace
User says "search the web"Claude uses web search
User provides a URLWeb fetch can retrieve the full page
Localised requestsClaude may use location inferred from IP

04What do Claude's search answers look like?

Anthropic says Claude processes multiple sources and replies with direct citations, source links and relevant quotes where appropriate (Anthropic). Image results are powered by Bing and include source links.

Our view: because responses can include quotes, pages with crisp, quotable sentences (definitions, facts, numbers with dates) are easier to cite.

05What content does Claude cite well?

Our view, based on the answer format above:

  • Direct answer in the first paragraph
  • Short, self-contained statements that make sense when quoted
  • Dates on facts and pages
  • Tables for comparisons and specs
  • Authorship and evidence for claims
  • No content hidden behind login walls or challenges

06How do you test Claude visibility?

  1. Turn on web search (or use the new experience) and add "search the web" to prompts for consistency.
  2. Run 20–50 buyer prompts (prompt library).
  3. Record citations, quoted text and whether your domain appears.
  4. Check logs for Claude-SearchBot and Claude-User hits and status codes.
  5. Re-run monthly.
PromptSearched?Your domain cited?Quoted text
"best AP automation for mid-sized companies"YesNo—

Ledgerline example; Ledgerline is fictional.

07What doesn't help?

  • Blocking Anthropic IPs instead of using robots.txt
  • Blocking all three bots and expecting citations
  • Long, unstructured pages without clear statements

08How do you check whether Claude can reach you?

Work through this in order:

  1. robots.txt: confirm no rule for * or a list of AI agents disallows Claude-SearchBot or Claude-User. Check each subdomain's own file, as Anthropic asks for rules per subdomain (Anthropic).
  2. Server logs: search for the Claude-SearchBot and Claude-User user agents. Note status codes; repeated 403 or 429 responses suggest a firewall or rate limit is blocking them.
  3. Bot challenges: if your CDN shows a challenge page to unknown bots, Anthropic's bots will not bypass it (Anthropic).
  4. Fetch test: paste a URL into Claude with web search on and ask for a summary. If Claude can't read it, fix access before working on content.

Our view: most "Claude never cites us" problems we see start with access, not content quality. Fix access first, because content work can't help a page Claude can't read.

09From MentionX data: what search-backed assistants citeOriginal data

Method. Read-only aggregate queries over MentionX scans from 22 August to 29 October 2026 (UTC): 2,217 engine runs (2,154 completed) across 18 anonymised projects and six engines.

We don't collect Claude answers, so we can't report Claude-specific rates. What we can show is how the six engines we track behaved, which gives a baseline for what to expect from any assistant that searches the web.

EngineBrand mentionedAnswers with no citations
Gemini47.3%50.7%
OpenAI42.9%30.2%
Perplexity31.5%4.1%
AI Mode31.3%12.6%
Copilot26.0%47.0%
DeepSeek24.0%99.0%

Brand mention rates ranged from 24.0% (DeepSeek) to 47.3% (Gemini). The share of answers without citations ranged even more widely, from 4.1% on Perplexity to 99.0% on DeepSeek. Across all engines, the page types cited most were:

Cited URL typeShare of 9,750 citations
Blog or article19.6%
Homepage10.0%
Review page2.7%
Docs2.6%
Pricing1.9%
Wikipedia1.6%

Our view: these differences are why you can't assume one engine's results carry over to another. Test Claude separately, with its own prompt set and log, rather than inferring from ChatGPT or Perplexity.

Limits. A convenience sample of mostly B2B software projects, run through engine APIs, which can behave differently from the consumer apps. Perplexity, Copilot and DeepSeek figures come from a single day (26 October 2026), so treat them as a snapshot, not a trend.

10Worked example: Ledgerline checks Claude access

Ledgerline is a fictional accounts-payable tool. Figures are illustrative.

Ledgerline never appears as a source when Claude searches "best AP automation for mid-sized companies".

Week 1: the team finds a robots.txt block list copied from a blog post. It disallows ClaudeBot, Claude-SearchBot and Claude-User together. They decide to keep blocking ClaudeBot, which relates to training, and allow the other two.

Week 2: server logs show Claude-SearchBot now receiving 200 responses, except on the docs subdomain, which has its own robots.txt with the same old rules. They fix that file too.

Weeks 3 to 8: they rewrite the AP automation guide to open with a direct answer and a dated feature table, then run 20 prompts three times every two weeks, logging cited URLs and the quoted text.

Anthropic documents ClaudeBot, which collects content that may be used for training, separately from Claude-SearchBot and Claude-User, which fetch pages for search and user requests. That lets you make two separate decisions. Blocking training is a legitimate choice. Blocking search access means Claude can't fetch your pages when a user's question needs current information, so it can't cite them.

12Which pages should you check first?

Start with the pages most likely to answer buyer questions: product overviews, comparison pages, pricing, integration pages and your most useful guides. Across the engines we track, blog posts and articles were cited most (19.6% of citations), followed by homepages (10.0%); pricing pages made up 1.9% and docs 2.6%. Make sure the guides that get cited also state your key product facts in plain sentences, so a quote from them can support a recommendation.

13How do prompt types affect results?

Across the engines we track, the tracked brand was named far more often for some intents than others:

IntentBrand mentionedShare of voice
Brand-specific100%64.3
Comparison91.9%44.6
Recommendation29.2%26.8
Best-of26.6%22.7
Problem5.7%7.7

When testing Claude, include each intent. Brand-specific prompts test accuracy. Comparison prompts against Competitor A to D test positioning. Unbranded best-of and problem-led prompts show whether you're considered at all, which is where the real gap usually is.

14How should you report Claude results?

Report the share of answers that searched the web, the share citing your domain, and the share naming your brand, each with the number of answers. Add the quoted text when Claude quotes you, because it shows which sentences are doing the work. Answers vary: across the engines we track, results changed between runs for 11 of 28 prompt-engine pairs we retested, so one run per prompt isn't a result.

15How long until changes show?

Access fixes show once Claude's bots can fetch the page, which you can confirm in logs. Content changes depend on when the page is next fetched for a relevant question. We have no Claude timing data, so plan to re-test every two weeks for at least two months before drawing conclusions.

16What makes a page easy for Claude to quote?

Anthropic says Claude replies with direct citations, source links and relevant quotes where appropriate. A quote needs a sentence worth quoting. That means:

  • an answer in the first lines under each heading, written so it makes sense on its own;
  • specific, dated facts rather than general claims;
  • product names and categories stated plainly, so the quote carries your name with it;
  • no key facts locked in images, PDFs or tabs that need scripts to open.

Our view: the quote test is a useful editing tool. Read each section's opening sentence alone and ask whether it would make sense in someone else's answer. If it relies on the paragraph before it, rewrite it.

17Who should own Claude work?

SEO or web operations owns robots.txt, CDN rules and log checks across every subdomain. Content owns the pages that answer buyer questions. Product marketing owns the facts. One person runs the test log, using the same prompts, settings and cadence each cycle, so results can be compared over time.

18Common mistakes

  • Copying a "block all AI bots" list without checking what each bot does.
  • Fixing the main robots.txt but not those on subdomains.
  • Testing with search switched off, then concluding Claude ignores your site.
  • Reporting one run per prompt as a result.

19How does Claude fit in a multi-engine programme?

Treat it as one engine in a shared process. The access checks, content standards and test method are the same as for ChatGPT, Perplexity and Gemini; only the bots and the settings differ. Keep one prompt set across engines so you can compare like with like, and one log format so results sit side by side. Then decide where to spend effort based on where your buyers actually ask questions, which a "How did you hear about us?" field that lists AI assistants can help you find out.

20Common questions

Does blocking ClaudeBot stop Claude citing me? Anthropic documents ClaudeBot separately from Claude-SearchBot and Claude-User. Blocking only ClaudeBot shouldn't stop search access, but check your rules for each bot.

Does Claude always search the web? No. Anthropic describes when Claude searches; many answers come from the model without searching.

Can I see Claude traffic in analytics? Visits from cited links may show claude.ai as the referrer; add it to your AI assistants channel.

Should I use Crawl-delay for Claude's bots? Anthropic says its bots support it. Use it only if crawling causes load problems, since slowing access can delay how quickly new pages are fetched.

21Checklist

  • Bot policy per Anthropic agent
  • Rules on every subdomain
  • No CAPTCHA in front of public content
  • Quotable, dated content
  • Monthly tests and log checks

Questions people ask

3 answers

How do I rank in Claude?

Allow Claude-SearchBot and Claude-User in robots.txt, publish clear pages that answer specific questions with current facts, and earn coverage on authoritative sites Claude is likely to cite.

Does blocking ClaudeBot remove me from Claude's answers?

ClaudeBot is for training data. Anthropic says blocking it excludes future content from training. Search visibility depends on Claude-SearchBot and Claude-User.

Does Claude always search the web?

No. Claude searches when a question benefits from current information, or when web search is on and the user asks it to.

Sources

  1. Anthropic, Does Anthropic crawl data from the web? (7 Apr 2026); Enable and use web searchsupport.claude.com
Written and reviewed by Dipuck JonesHow we research and testReport an error