What Is llms.txt? Complete Guide

llms.txt is a proposed plain-text file, written in Markdown and placed at `/llms.txt`, that gives large language models a short, curated guide to your site: what it is and which pages matter most. It was proposed at llmstxt.org in 2024 to help AI tools use websites at inference time. It's optional and not a ranking factor, and it doesn't control crawling.

On this page

• Why was llms.txt proposed?
• What goes in an llms.txt file?
• Product
• Learn
• Optional
• How do you write good link notes?
• How is llms.txt different from robots.txt and sitemap.xml?
• Which AI systems read llms.txt?
• From MentionX data: do AI answers cite llms.txt?
• Worked example: Ledgerline decides on llms.txt
• Is llms.txt worth adding?
• What are the common mistakes?
• How do you generate one without writing it by hand?
• Where does llms.txt fit in your priorities?
• What are the less obvious mistakes?
• Who maintains llms.txt?
• How do you check it's working?
• Common questions
• Sources

Checked against llmstxt.org and vendor documentation on 30 October 2026. "Our view" marks MentionX opinion. The worked example uses a fictional company.

Why was llms.txt proposed?

The proposal's argument is practical. Context windows are too small for most websites, and converting HTML pages full of navigation, ads and scripts into clean text is hard and error-prone. A single Markdown file that summarises the site and links to key pages, ideally to Markdown versions of them, makes it easier for a model to find the right information (llmstxt.org).

It's aimed mainly at inference time, when a user asks an assistant something and the assistant reads your site. It isn't designed as a training opt-in or opt-out.

What goes in an llms.txt file?

The proposal defines a simple structure (llmstxt.org):

1. An H1 with the project or site name. This is the only required section.
2. A blockquote with a short summary and the key information needed to understand the rest.
3. Optional free-text sections with more detail (no headings).
4. H2 sections containing lists of links, each a Markdown link, optionally followed by a colon and a note.
5. An "Optional" H2 section for links that can be skipped when context is short.

The proposal also suggests offering Markdown versions of pages at the same URL with .md appended.

A short example for a SaaS marketing site:

# MentionX

> MentionX is an AI visibility platform that tracks how ChatGPT, Perplexity, Gemini and Google AI features mention, cite and recommend brands.

## Product
- [How it works](https://example.com/how-it-works.md): prompt tracking, engines covered, scan frequency
- [Pricing](https://example.com/pricing.md): plans and limits, updated monthly

## Learn
- [What is GEO?](https://example.com/guides/generative-engine-optimization.md): plain-English guide

## Optional
- [Changelog](https://example.com/changelog.md)

The step-by-step build is in How to Create an llms.txt File.

How do you write good link notes?

The note after each link is where most files are weakest. A model uses the note to decide whether a page answers the question in front of it, so describe what's on the page, not how good it is.

Keep notes to one line. Use the nouns your buyers use. Include facts that change, such as plan names and supported systems, only if the file is regenerated when they change.

Write the summary blockquote the same way: say what the product is, who it's for and the one or two facts that set it apart, in two or three plain sentences. Avoid claims like "leading" or "best-in-class". A model can't verify them, and they add nothing it can use.

How is llms.txt different from robots.txt and sitemap.xml?

Blocking or allowing an AI crawler is always done in robots.txt (and your CDN). A full comparison is in llms.txt vs robots.txt.

Which AI systems read llms.txt?

Be careful with claims here. As of 30 October 2026:

• Google says there's no need for "new machine readable files, AI text files, or markup" to appear in AI Overviews or AI Mode (Google Search Central).
• OpenAI, Anthropic and Perplexity document their crawlers and robots.txt controls (OpenAI, Anthropic, Perplexity), but their crawler documentation doesn't present llms.txt as a requirement for appearing in answers.
• Developer tools and coding assistants are where it's most visible: many documentation platforms generate llms.txt automatically, and some assistants can be pointed at a docs site's llms.txt.

If a vendor tells you llms.txt will raise your AI rankings, ask for evidence.

From MentionX data: do AI answers cite llms.txt?

Method. We counted every citation URL returned in completed MentionX runs from 22 August to 29 October 2026 (UTC): 9,750 citations from 2,154 runs across 18 anonymised projects and six engines. We stripped query strings and checked the path ending.

In the non-Gemini citations we could classify (n = 5,917), blog, guide and resource paths made up 19.6% and homepages 10.0%. These are ordinary HTML pages.

Limits. About 69% of Gemini's cited URLs are Google redirect links that hide the destination, so a few could resolve to Markdown files. A citation also isn't the same as a read: an assistant could read llms.txt to find a page, then cite the page. Our prompts are buyer questions about B2B software, not coding questions, where docs files matter more.

Our view: the data says llms.txt isn't something answer engines show to users. It doesn't say the file is useless. Its likely value is upstream, in helping an agent or coding assistant find the right page quickly.

Worked example: Ledgerline decides on llms.txt

Ledgerline is a fictional accounts-payable tool. It has a marketing site with 60 pages and a docs site with 400 pages, including an API reference used by customers' developers.

The team scores each site against three questions:

They switch on generation for the docs site that week and park the marketing site file until crawler access and the comparison pages are fixed. A month later the docs logs show regular requests to /llms.txt, mostly from developer tools. The marketing team adds a 12-link file in an afternoon at the end of the quarter, because the cost is low, without expecting a visibility change. Illustrative: figures describe the method, not real logs.

Is llms.txt worth adding?

What are the common mistakes?

• Using llms.txt to try to block AI. It can't; use robots.txt.
• Dumping every URL. The point is curation. Keep it to the pages that answer the most important questions.
• Letting it go stale. Old pricing links do more harm than no file. Add it to your release checklist.
• Writing marketing copy in the summary. Write a plain, factual description a model can repeat accurately.

How do you generate one without writing it by hand?

Most sites shouldn't maintain the file manually. Common options, each documented by its maker:

Whichever route you choose, edit the output. Generators tend to include too many pages and write flat summaries. The useful part of the file is the human decision about which dozen pages matter and what each one answers.

Where does llms.txt fit in your priorities?

Our view, for a typical B2B brand trying to be mentioned and recommended in AI answers, in order of likely impact:

1. Crawler access. Search crawlers and Googlebot can fetch your key pages, and your CDN isn't blocking them.
2. Clear pages that answer buyer questions, such as pricing, comparisons, integrations and use cases, in text the crawler can read.
3. Third-party coverage: reviews, lists, community threads and press. In our data, comparison prompts named the tracked brand in 91.9% of answers and problem-type prompts in 5.7%, so your category presence outside your own site does most of the work.
4. Structured data that states who you are consistently.
5. llms.txt, especially for docs.

Item 5 is cheap, so it's fine to do it early. Just don't count it as progress on items 1 to 4.

What are the less obvious mistakes?

• Linking to pages behind a login or a cookie wall. A tool reading the file can't get past them, so the link wastes space.
• Pointing to `.md` URLs that don't exist. The proposal suggests Markdown copies, but a link that returns 404 is worse than linking to the HTML page.
• Using a different product name from your site and profiles. Variations make it harder for a model to connect the file to your brand.
• Blocking the file at the CDN. Some bot-protection rules challenge any non-browser request, including requests for /llms.txt. Fetch it with a plain command-line request to check.
• Measuring success by rankings or mentions. Nothing public suggests the file changes either. Measure what it can influence: requests in your logs, and whether tools that use it land on the right pages.

Who maintains llms.txt?

Give it an owner, usually whoever owns the docs or the website CMS. Good practice:

• Generate it from your CMS or docs build rather than editing by hand, so new key pages are included automatically.
• Review the summary blockquote whenever positioning changes, so it matches the homepage and your Organization structured data.
• Keep a short changelog in your repo noting what changed and why.

How do you check it's working?

1. Visit yoursite.com/llms.txt and confirm it returns 200 with text/plain or text/markdown.
2. Check every link resolves.
3. Search server logs for requests to /llms.txt and note the user agents. That's your only real evidence of use.

Common questions

Should llms.txt appear in Google results? It doesn't matter much either way. If you'd rather it didn't, send an X-Robots-Tag: noindex HTTP header for the file (Google). Don't block it in robots.txt, or tools that respect robots.txt can't read it.

Does the file need to be in English? Write it in the main language of the pages it links to. A multilingual site can link to each language's key pages under separate H2 sections.

Does llms.txt replace a sitemap? No. A sitemap lists every URL for search engines; llms.txt lists a curated few with context. Keep both.

What's llms-full.txt? A convention used by several docs platforms: one file containing the full text of the docs in Markdown, so a tool can load everything at once. It's not part of the original proposal's required structure.

Can I put it in a subfolder? The proposal places it at the root, /llms.txt. Some sites also publish one per docs section; keep a root file that links to them.

How long should the file be? For most sites, 10 to 40 links. A docs site can go further, but put the long tail in the Optional section or an llms-full.txt file.

Will it hurt anything? Not if it's accurate. The risk is a stale file that points tools to old pricing or retired features.

Sources

• llmstxt.org, the llms.txt proposal
• Google Search Central, AI features and your website
• IETF, RFC 9309: Robots Exclusion Protocol
• OpenAI, Overview of OpenAI crawlers; Anthropic, crawler FAQ; Perplexity, crawlers
• Mintlify, llms.txt; Yoast, llms.txt feature; WordPress.org, Website LLMs.txt plugin; Firecrawl, create-llmstxt-py
• Cloudflare, Markdown for Agents