llms.txt vs robots.txt

robots.txt tells crawlers which URLs they may or may not fetch. llms.txt gives language models a curated Markdown summary of your site and its most useful pages. One is access control, standardised as RFC 9309. The other is an optional guide proposed at llmstxt.org. You can use both, but only robots.txt changes what compliant crawlers are allowed to do.

On this page

• What does robots.txt do?
• What does llms.txt do?
• How do the files work together during a request?
• What about noindex and other page-level controls?
• How do they compare side by side?
• From MentionX data: where the effort pays off
• Which setup fits your situation?
• How do you keep the two files consistent?
• Worked example: Ledgerline lines up both files
• Which mistakes cause the most damage?
• Common questions
• How do you test robots.txt changes before release?
• Who should own each file?
• What's changing around both files?
• Where should you start?
• Sources

Checked against RFC 9309, llmstxt.org and vendor crawler documentation on 30 October 2026. "Our view" marks MentionX opinion. Ledgerline is fictional.

What does robots.txt do?

robots.txt sits at your domain root and lists rules per user agent, using Allow and Disallow paths. The Robots Exclusion Protocol was standardised by the IETF as RFC 9309 in 2022. Google explains that robots.txt manages crawler traffic and isn't a way to keep a page out of Google; for that you use noindex (Google: robots.txt intro; noindex).

AI companies document their user agents and say how they respect robots.txt: OpenAI for GPTBot, OAI-SearchBot and ChatGPT-User (OpenAI), Anthropic for ClaudeBot and related agents (Anthropic), Perplexity for PerplexityBot (Perplexity), and Google for Google-Extended (Google).

What does llms.txt do?

llms.txt is a Markdown file at /llms.txt with an H1 name, a blockquote summary and H2 sections listing key links with short notes (llmstxt.org). Its goal is to help models find and use the right content at inference time. It contains no directives.

How do the files work together during a request?

Follow one AI search request through your site to see where each file acts:

1. The crawler reads robots.txt. Before fetching anything, a compliant crawler checks the rules for its user agent. If the path is disallowed, it stops. Google says it generally caches robots.txt for up to 24 hours, so changes take time to apply (Google).
2. Your CDN or firewall decides whether to serve the request. This layer sits outside both files. A bot rule can return 403 or a challenge page even when robots.txt allows the agent.
3. The crawler fetches the page. It reads the HTML your server returns. Content added later by JavaScript may be missed (Google).
4. An agent may read llms.txt for orientation. If a tool supports the file, it reads the summary and links to choose which pages to fetch next. Each of those fetches goes back through steps 1 to 3.

llms.txt can only point to pages. It can't open a path that robots.txt or the CDN has closed.

What about noindex and other page-level controls?

robots.txt controls fetching, not indexing. A blocked URL can still appear in Google results without a snippet if other pages link to it. To keep a page out of results, Google says to allow crawling and use a noindex meta tag or header instead (Google). For Google's AI features, the same snippet controls apply: nosnippet and related tags limit what can be shown (Google).

How do they compare side by side?

From MentionX data: where the effort pays off

Method. Read-only aggregate queries over completed MentionX runs from 22 August to 29 October 2026 (UTC): 2,154 runs and 9,750 citations across 18 anonymised projects and six engines.

Engines such as Perplexity and AI Mode cite on almost every answer, and those citations come from pages their crawlers or indexes could fetch. That's the job robots.txt controls. Nothing in our data shows answers citing llms.txt.

Limits. About 69% of Gemini's cited URLs are Google redirect links that hide the destination. A tool could read llms.txt without citing it. The sample covers B2B buyer questions, not developer questions.

Our view: treat robots.txt as required maintenance and llms.txt as an optional extra that's most useful for docs.

Which setup fits your situation?

Scenario 1: You want maximum AI visibility. robots.txt allows Googlebot, Bingbot, OAI-SearchBot, ChatGPT-User, PerplexityBot and Claude-SearchBot; optionally allows training bots too. Add llms.txt if you have docs or a help centre.

Scenario 2: Visible in AI search, but no training use.

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Google-Extended
Disallow: /

llms.txt is still fine to publish; it doesn't grant training access to anyone.

Scenario 3: Paywalled or members-only content. Use authentication, not robots.txt, to protect content; robots.txt is public and only advisory. Keep gated URLs out of llms.txt.

Scenario 4: You want AI to stay away entirely. Disallow every documented AI agent in robots.txt, back it with CDN bot rules, and don't publish llms.txt. Accept that you will rarely be cited.

How do you keep the two files consistent?

Run this check each quarter:

1. List every URL in llms.txt.
2. Test each against robots.txt for the agents you care about (OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot).
3. Any URL listed in llms.txt but disallowed for those agents is a contradiction. Fix one file or the other.
4. Confirm neither file references staging domains or old paths.

Worked example: Ledgerline lines up both files

Ledgerline is a fictional accounts-payable tool. Its policy is Scenario 2: visible in AI search, no training use. A quarterly check of both files finds three problems.

The first finding mattered most. Comparison prompts named the tracked brand in 91.9% of answers in MentionX data (n = 272), against 26.6% for best-of prompts. Pages that answer "X vs Y" questions are among the most valuable a B2B site has, and robots.txt had been hiding them from every AI search crawler.

Which mistakes cause the most damage?

• A blanket `User-agent: / Disallow: /`** left over from staging. It blocks every compliant crawler, AI or not.
• Blocking GPTBot and assuming ChatGPT search is blocked too. OpenAI treats them as separate agents (OpenAI).
• Blocking Google-Extended to "leave AI Overviews". Google says Googlebot rules control Search including its AI features (Google).
• Using robots.txt to hide sensitive pages. It's public; anyone can read the paths you list.
• Putting `Disallow` lines inside llms.txt. They do nothing there.

Common questions

If I block a crawler in robots.txt, can it still use my llms.txt? A compliant crawler won't fetch blocked paths. If /llms.txt itself is allowed, it can read the file but not the blocked pages it links to.

Do I list llms.txt in robots.txt? There's no directive for it. robots.txt can list sitemaps with Sitemap:, but nothing equivalent exists for llms.txt.

Does llms.txt tell AI companies I consent to training? No. It says nothing about permissions. Training preferences are set per agent in robots.txt, such as GPTBot, ClaudeBot and Google-Extended.

Is a missing robots.txt a problem? Not for access. Under RFC 9309, if the file returns 404, crawlers may fetch everything. A missing file just means you have no way to set rules per agent, and server errors (5xx) on robots.txt can make Google pause crawling.

Can llms.txt and robots.txt disagree on purpose? Yes, for example listing a page for tools that ignore robots.txt. It's rarely worth the confusion. A file that contradicts your crawler rules makes audits harder and suggests a mistake to anyone who reads both, including your own team a year later.

Which file do user-initiated fetchers follow? Neither, necessarily. OpenAI and Perplexity say robots.txt may not apply when a user asks for a page (OpenAI, Perplexity). Authentication is the only reliable way to keep content private.

How do you test robots.txt changes before release?

• Check the syntax. One misplaced line, such as Disallow: / under the wrong group, can block a whole site. Use a robots.txt parser in your build, or Search Console's robots.txt report for Google.
• Test named agents, not just ``.** A crawler follows only the most specific group that matches it (Google). Test the exact tokens you care about.
• Fetch as the agent. Request a key page with each user agent string and confirm your CDN returns 200, not a challenge. This won't prove the real bot gets through, because CDNs also check IPs, but it catches simple user-agent rules.
• Check logs after release. Within a week you should see the allowed agents receiving 200s on the paths you opened.

Who should own each file?

robots.txt usually belongs to the SEO or web platform team, because a single bad line can remove a site from search. Changes should go through code review and be tested before release. llms.txt can sit with whoever owns docs or content. Agree on one rule between the two owners: any change to which AI agents are allowed is a policy decision, written down with a date, not a quick edit.

What's changing around both files?

Neither file stands still, so check the primary sources each quarter.

• Training preferences. The IETF AI Preferences (AIPREF) work aims to give sites a standard way to state how content may be used, such as for training, rather than only which paths can be fetched. Google says it intends to support the AIPREF training preference signal in early 2027 (Google). If that happens, training choices may move from per-agent robots.txt groups to a single signal.
• New agents. Companies add user agents as they launch products. OpenAI's list now includes an agent that checks ChatGPT ad landing pages (OpenAI). Each new agent needs a policy decision.
• Markdown for agents. Instead of a separate file, some CDNs can serve Markdown versions of pages to agents that ask for them through content negotiation. Cloudflare documents this as Markdown for Agents (Cloudflare). It does a similar job to the proposal's .md copies, and it still depends on robots.txt and CDN access.
• llms.txt adoption. Support is growing in developer tools and docs platforms. There's still no public statement from Google, OpenAI, Anthropic or Perplexity that their answer engines use it to choose sources.

Our view: the gate (robots.txt, CDN rules and, later, AIPREF signals) will keep getting more detailed. The map (llms.txt or Markdown for agents) will matter more as agents do more tasks on users' behalf. Neither replaces clear public pages and third-party coverage.

Our view: treat robots.txt as the file that matters today, because crawlers act on it. Treat llms.txt as a low-cost extra until engines state that they read it.

Where should you start?

Get robots.txt right first; it affects every engine. Then decide whether llms.txt helps your users. Step-by-step guides: Configure robots.txt for AI crawlers and Create an llms.txt file.

Sources

• IETF, RFC 9309: Robots Exclusion Protocol
• Google Search Central, Introduction to robots.txt; Block indexing with noindex; AI features; Common crawlers
• llmstxt.org
• Google, How Google interprets the robots.txt specification
• Google, JavaScript SEO basics
• Cloudflare, Markdown for Agents
• OpenAI, crawlers; Anthropic, crawler FAQ; Perplexity, crawlers