How to Opt Out of AI Training and Stay in AI Search
Add two lines to robots.txt to say no to AI training and still appear in AI answers. What the lines mean, who follows them today and what to keep.
You can now tell AI companies not to train on your content with two lines in robots.txt, and still show up in AI answers. The two lines are `User-Agent: __usage_ai-training` and `Disallow: /`. They only cover training. They do not stop AI search crawlers from reading your pages ([IETF draft](https://www.ietf.org/archive/id/draft-kelley-aipref-training-preference-signal-00.html)). Anthropic says it already honors the signal; other providers have announced later dates, so keep your existing crawler rules for now.
- One group states your training preference to every provider that adopts the signal.
- It separates "may you train on this?" from "may you crawl this?". Visibility in AI answers depends on the second.
- A wildcard group (
User-Agent: *) does not count as this signal. You must name the token. - Adoption is uneven today. Keep GPTBot, ClaudeBot and Google-Extended rules until each provider confirms.
Checked 17 November 2026. The signal is defined in an Internet-Draft, which is a proposal and not yet a standard; it expires on 3 May 2027 unless updated. "Our view" marks MentionX opinion.
01What does the signal look like?
The draft's own example:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/| Line | Meaning under this token |
|---|---|
User-Agent: __usage_ai-training | The rules below are about AI training, not about a crawler |
Disallow: / | Preference against AI training for the whole site |
Allow: /press-kit/ | Preference in favour of AI training for that folder |
Paths match as they do elsewhere in robots.txt, with the longest match winning. The two underscores at the start are part of the name.
02What counts as "AI training"?
The draft uses the working group's definition: using content to modify the learned parameters of a generative AI model. It does not address search, answering a user's question from retrieved pages, or fetching a page because a user asked. Those are outside the signal.
That is the point for AI visibility. Being cited in an answer depends on search crawlers reaching your pages, and on fetches made when a user asks, which this signal does not touch.
03What does it not do?
| It does not | What to use instead |
|---|---|
| Stop any crawler from fetching pages | A group that names the crawler, such as User-agent: GPTBot |
Apply if you only have User-Agent: * rules | Add a group that names __usage_ai-training |
| Reach providers that have not adopted it | Their own crawler tokens |
| Remove content already collected | Nothing in this draft; it applies to the robots.txt in effect when a page is fetched |
| Enforce anything | It expresses a preference; the draft notes private agreements can override it |
If no group names the token, you have expressed no preference either way.
04Who honors it, and from when?
| Provider | Position | Source |
|---|---|---|
| Anthropic | Recognises the signal as of 30 October 2026, rolling out across its services; a rule naming ClaudeBot takes priority over the general signal | Anthropic Help Center |
| Says it intends to support the signal in early 2027 | Google crawler documentation | |
| Microsoft (Bing, Copilot) | Will follow the signal from 1 February 2027, as it crawls pages. A setting on a single page overrides robots.txt | Microsoft Bing |
| OpenAI | Announced on 30 October 2026 that it will honor the signal, reported as "in the coming months"; says existing crawler controls stay | Check OpenAI's crawler page |
We read the draft and the Anthropic, Google and Microsoft pages ourselves. The OpenAI row rests on its announcement as reported; confirm the date before you rely on it. Bing has also added a separate setting for AI answers, covered in Bing AI settings: stay in Copilot answers.
05Does opting out of training cost you AI visibility?
Under the signal as written, no. But providers differ in how they connect training to other uses, so check each one you care about:
- Anthropic runs separate bots: ClaudeBot for training, Claude-SearchBot and Claude-User for search and user requests. Opting out of training leaves the other two alone. See ClaudeBot and Claude-SearchBot.
- Google controls training and Gemini app grounding through Google-Extended, which is separate from Google Search, AI Overviews and AI Mode. See Google-Extended controls.
- OpenAI separates GPTBot (training) from OAI-SearchBot (ChatGPT search).
- Meta separates Meta-ExternalAgent (training) from Meta-WebIndexer (Meta AI search). See how to rank in Meta AI.
Our view: for a brand that wants to be recommended by AI tools, the safe pattern is "no to training if you wish, yes to search and user requests". Blocking everything with "AI" in the name is the mistake that costs visibility.
06What should your robots.txt say today?
If you want to opt out of training and stay visible in AI answers:
# Shared training preference
User-Agent: __usage_ai-training
Disallow: /
# Keep provider specific training rules until each adopts the shared signal
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: meta-externalagent
Disallow: /
# Search and user-request agents stay allowed
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: meta-webindexer
Allow: /Note the Google-Extended line: Google says that token also governs grounding in the Gemini app, so disallowing it has an effect beyond training. Decide that one deliberately.
If you are happy for your content to be used in training, you need no new lines. Saying nothing expresses no preference.
07How do you check it is in place?
- Open
yourdomain.com/robots.txtin a browser and find the group. - Confirm it is spelled exactly, with two leading underscores.
- Confirm the agents used for search and for user questions are still allowed.
- Repeat for each subdomain; robots.txt applies per host.
- Put a date in your calendar to check provider support again each quarter and remove duplicate rules once they are no longer needed.
08Who should decide?
This is a business and legal choice, not a technical one. Marketing should bring one fact to that conversation: visibility in AI answers runs through the agents used for search and for user questions, and the training preference can be set without touching them. More in blocking and allowing AI crawlers.
09Checklist
- Decide your training preference with legal.
- Add the
__usage_ai-traininggroup if you are opting out. - Keep provider specific training rules for now.
- Confirm the agents for search and user questions are allowed.
- Check every subdomain.
- Review provider support each quarter.
10Related
Questions people ask
5 answersWhat is the AI training opt out line in robots.txt?
+
It is a short block you add to robots.txt that tells AI companies whether they may use your content to train their models. It was proposed on 30 October 2026 and uses the name __usage_ai-training.
Does the AI training opt out block crawlers?
+
No. The draft says the token is not a crawling directive. It states a preference about use, and crawlers still follow the rules written for their own names.
Will opting out of AI training remove my site from AI search answers?
+
The signal as defined covers training only, not search or answering questions. Check each provider, though. Google ties its training control to grounding in the Gemini app.
Which AI companies honor the signal today?
+
Anthropic says it has honored it since 30 October 2026. Google says it intends to support it in early 2027. Microsoft and OpenAI have announced support with later start dates.
Should I remove my GPTBot and ClaudeBot rules now?
+
No. Keep them. Most providers have not switched the shared signal on yet, and a rule that names a specific crawler still applies.
Sources
- IETF, Machine-Readable Training Preference Signal (Internet-Draft, 30 October 2026)ietf.org
- Anthropic Help Center, Does Anthropic crawl data from the web, and how can site owners block the crawler?support.claude.com
- Google, Google's common crawlersdevelopers.google.com