How Claude Chooses Sources and How to Get Your Brand Cited

#01Why should marketers care how Claude picks sources?

Until now this track has covered Claude as a tool you use. This lesson flips it around: Claude is also a place where your buyers ask questions, and its answers can cite your pages, a competitor's or neither.

Using Claude every day gives you an advantage here. You've seen which sources it cites for questions in your field, how it phrases answers and when it searches rather than answering from what it already knows. That's direct evidence about how an AI engine treats your category.

Our view: marketers who use Claude well tend to understand AI visibility faster than those who only read about it. Every research session is also a small experiment in what gets cited.

#02How does Claude decide to search the web?

In the Claude apps, web search lets Claude pull current information. Anthropic says that when you ask about topics that benefit from current information, Claude "invokes a search tool to inform and ground its generated responses", and that every response that uses it includes citations (Anthropic).

For questions where training knowledge is enough, Claude may answer without searching. Questions about recent events, prices, products, comparisons and anything that changes tend to trigger searches. That matters for marketers: the questions buyers ask about vendors, pricing and alternatives are exactly the ones likely to send Claude to the web.

Developers can also use web search through the API. Anthropic's documentation describes a web search tool that returns cited sources and lets developers limit or block domains (Claude Platform docs). So the same search behaviour can appear inside other companies' products built on Claude, not only in Claude's own apps.

#03What are Anthropic's crawlers, and which should you allow?

Anthropic's help centre lists three robots (Anthropic):

Anthropic says its robots honour robots.txt, respect anti-circumvention measures such as CAPTCHAs and support the non-standard Crawl-delay directive.

The practical decision is usually about ClaudeBot. Some organisations choose to block training while allowing Claude-User and Claude-SearchBot so their pages can still be found and cited in answers. Whether to allow training is a business and legal decision; if you're unsure, involve whoever owns your content licensing. We cover the details in ClaudeBot and Claude-SearchBot.

Check your robots.txt and any firewall or bot-management rules. Blocking at the network level, sometimes by default in security tools, has the same effect as a robots.txt block, and marketing teams often don't know it's happening.

#04What kinds of pages get cited?

Anthropic doesn't publish a ranking formula for which sources Claude cites, so be wary of anyone who claims to know it exactly. What you can observe, by asking Claude questions in your category and reading the citations, is which pages it uses.

Pages that tend to be easy for any AI engine to use share some traits:

• They answer one clear question, with the answer near the top rather than after a long introduction.
• Their headings are questions or clear topics, so a section can be lifted and understood on its own.
• Their facts are specific and sourced: dates, numbers with sources, named standards.
• They're accessible: the main content is in the HTML, not hidden behind scripts, logins or heavy interstitials.
• They're current, with dates and updates where the topic changes.

We go deeper in How Claude picks sources and citations and How to rank in Claude.

#05How can you use Claude to audit your own visibility?

Turn on web search and ask the questions your buyers ask. Use a fresh chat with memory and project context out of the way, so your own files don't shape the answer. Anthropic offers incognito chats that aren't saved to history or memory, which suit this.

I'm researching [category] for a [company type]. What are the
main options, and how do they compare on [criteria]? Cite
your sources.

Run a set of ten to twenty real questions and record, for each: whether Claude searched, which sources it cited, whether your brand appeared, and how it was described. Repeat monthly. Answers vary between runs, so look for patterns rather than reacting to a single answer.

Then ask Claude about the sources themselves: "For the sources you cited, what made each useful for this question?" The answer isn't an official explanation of ranking, but it often highlights what your pages lack, such as a direct comparison, a pricing table or a recent date.

#06What should you change on your site?

Start with the pages that answer your buyers' most common questions. For each, check that the first two sentences answer the question directly, that the page is reachable by Claude-User and Claude-SearchBot, and that the facts are current and sourced. Add question-style H2s where they help a reader scan.

Then fill gaps. If Claude cites third-party reviews or comparisons for questions about your category because you have no page on the topic, write one that's genuinely more useful and accurate. Keep it honest: pages that overclaim tend to lose trust with readers and engines alike.

#07From MentionX data: how often Claude cites brands in your categoryOriginal data

Method. Read-only aggregate queries over MentionX scans from 22 August to 29 October 2026 (UTC): 2,217 engine runs (2,154 completed) across 18 anonymised projects and six engines.

MentionX tracks citations across AI engines, including Claude. The aggregates show which page types and domains Claude cites in each category, and how that compares with other engines:

Our view: manual checks show you what happens; tracking shows whether it's changing. Use both, and focus on the questions closest to a buying decision.

Limits. A convenience sample of mostly B2B software projects, run through engine APIs, which can behave differently from the consumer apps. Perplexity, Copilot and DeepSeek figures come from a single day (26 October 2026), so treat them as a snapshot, not a trend.

#08Worked example: Ledgerline's citation check

Ledgerline is a fictional accounts-payable tool. Figures are illustrative.

Ledgerline's content lead asks Claude twenty buyer questions in incognito chats with web search on. Claude cites Ledgerline for three; for questions about pricing and integrations it cites review sites and Competitor A.

She finds that the firewall blocks some AI user agents, which the web team fixes, and that the pricing page doesn't state prices in text. Ledgerline rewrites the integrations page around buyer questions and adds a dated pricing table. She reruns the questions monthly. Ledgerline is fictional and the details are illustrative.

#09Where do you go from here?

This lesson closes the first part of the Claude for Marketers track. You've seen how to use Claude in your daily work and how Claude treats your content. Run the visibility audit above, fix the access issues it reveals, and start one page improvement this week.

Next: Claude governance for marketing teams. For the engine view, read How to rank in Claude, or return to the Claude for Marketers track.

#10How does web fetch differ from web search?

There are two ways Claude reads the web in its apps. Web search runs queries and picks results. Web fetch, which Anthropic says works when web search is on, retrieves a specific page when someone gives Claude its URL. Both rely on Claude being able to reach and read your page.

Web fetch matters more than marketers often realise. Buyers paste vendor URLs into Claude and ask questions like "Summarise this pricing page" or "Compare these two products". If your page blocks Claude-User, or puts the important content behind scripts that don't render, Claude may give a thin or wrong summary of your offer while the competitor's page is summarised accurately.

Test it yourself. Paste your key pages into a chat with web search on and ask Claude to summarise them and answer a buyer's question from them. If it can't find your pricing, your main differentiator or your integrations, a buyer using Claude won't either.

#11How do training and search differ for your content?

It helps to separate two ways your content can affect Claude. The first is training: content collected by ClaudeBot may contribute to what future models know without searching. That knowledge has a cut-off date and isn't cited, so you can't easily see its influence. The second is retrieval: content found through search or fetch at the moment someone asks, which is current and usually cited.

For most marketing goals, retrieval is the part you can work on directly. You control whether the relevant robots can reach your pages, how clearly those pages answer questions and how current they are. Training influence is slower and harder to measure, and whether to allow it is a policy decision rather than a tactic.

#12How should marketing and web teams work together on this?

AI visibility touches several teams. Marketing owns the questions and content. The web or engineering team owns robots.txt, rendering and performance. Security often owns the firewall and bot-management rules that may block AI user agents without anyone in marketing knowing. Legal may have a view on training.

A short checklist shared between those teams prevents most problems: which Anthropic robots are allowed and why, how new bot-management rules are reviewed for their effect on AI user agents, and who checks monthly that key pages are still reachable. Claude can help draft that checklist and explain each item in plain language for non-technical colleagues.

#13How do you measure progress over time?

Manual checks are a good start but don't scale. Keep your question set fixed so results are comparable month to month, and record the same fields each time. Note changes you make to pages alongside the dates, so you can see whether citations follow.

Be patient and honest about what you see. Answers vary, search results change and engines update. A single month's improvement may be noise; a steady pattern over several months is more convincing. Treat Claude visibility as one input into your wider content and SEO programme, not a separate project with its own metrics nobody else understands.

When you report to leadership, show the question set, the trend and the changes you made, and be clear about what's uncertain. A careful, steady report builds more trust than a dramatic claim based on one good week, and it keeps the work funded long enough to show real results. Share the question set with sales and support too; they hear buyers' real questions every day and can suggest better ones.

#14What does a robots.txt for AI search look like?

Here are two common setups using Anthropic's documented user agents. Agree the policy with your web, legal and security teams before changing anything, and remember that robots.txt applies per subdomain.

Allow Claude to find and fetch your pages for answers, but opt out of training:

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

Allow everything except private areas:

User-agent: ClaudeBot
User-agent: Claude-User
User-agent: Claude-SearchBot
Disallow: /account/
Disallow: /checkout/

Anthropic notes that its robots honour robots.txt and also respect anti-circumvention technologies such as CAPTCHAs. That means a bot-protection challenge on your site can stop Claude-User even when robots.txt allows it.

#15How do you check whether Claude's robots can reach you?

Three checks, in increasing order of effort:

1. Ask Claude. In a chat with web search on, paste a key URL and ask it to quote the first paragraph and the page's main heading. If it can't, something is blocking or the content isn't in the HTML.
2. Check your server or CDN logs. Search for the user agents ClaudeBot, Claude-User and Claude-SearchBot and look at the status codes. Many 403 or challenge responses point to a firewall rule. Claude can help: export a sample of log lines (with IP addresses removed if policy requires) and ask it to summarise requests and status codes by user agent and path.
3. Review bot-management settings. Many CDNs and security tools have AI bot categories or one-click blocks. Check what's enabled and when it changed.

#16Which questions should be in your audit set?

A good question set mirrors how buyers actually use AI engines. Include a mix:

• Category questions: "What are the best [category] tools for [audience]?"
• Comparison questions: "[Your brand] vs Competitor A for [use case]"
• Problem questions: "How do I [problem your product solves]?"
• Evaluation questions: "Is [your brand] good for [use case]? What are its downsides?"
• Pricing and fit: "How much does [category] software cost for a team of 10?"

Get real wording from sales calls, support tickets and site search, not from your own marketing language. Keep the set to a size you'll actually rerun, usually 15 to 30 questions, and record results in a spreadsheet with the date, whether Claude searched, the cited URLs and how your brand was described.

#17What do you do with the results?

Sort cited sources into three groups: your own pages, third-party pages that mention you, and pages that don't mention you. Each calls for different work.

• Your pages cited: keep them current and accurate; they're working.
• Third parties cited, mentioning you: check that what they say is accurate. If it's out of date, contact the publisher with corrections.
• Pages cited that don't mention you: these show where the conversation happens without you. Ask whether you have a page that answers the question better, and whether those publishers would be interested in your data or perspective.

Avoid the temptation to write pages designed only to be cited. Pages written for readers, answering their questions clearly with evidence, are the ones that hold up as engines change.

#18Common questions

Does blocking ClaudeBot stop Claude citing my site? ClaudeBot is for training. Citations in answers rely on Claude-User and Claude-SearchBot; Anthropic says blocking those may reduce visibility.

Can I pay to be cited by Claude? Anthropic's documentation describes citations as coming from web search and page retrieval, and doesn't describe any paid option to be cited.

Why does Claude cite different sources each time? Searches and answers vary between runs. Test a set of questions repeatedly and look for patterns rather than single results.