# How to Opt Out of AI Training and Stay in AI Search

> Add two lines to robots.txt to say no to AI training and still appear in AI answers. What the lines mean, who follows them today and what to keep.

Source: https://mentionx.ai/learn/technical/ai-training-opt-out-robots-txt-signal  
Author: Dipuck Jones, founder of MentionX  
Updated: 17 November 2026  
Level: Practitioner

**You can now tell AI companies not to train on your content with two lines in robots.txt, and still show up in AI answers. The two lines are `User-Agent: __usage_ai-training` and `Disallow: /`. They only cover training. They do not stop AI search crawlers from reading your pages ([IETF draft](https://www.ietf.org/archive/id/draft-kelley-aipref-training-preference-signal-00.html)).** Anthropic says it already honors the signal; other providers have announced later dates, so keep your existing crawler rules for now.

*Checked 17 November 2026. The signal is defined in an Internet-Draft, which is a proposal and not yet a standard; it expires on 3 May 2027 unless updated. "Our view" marks MentionX opinion.*

> **Key takeaways**
>
> - One group states your training preference to every provider that adopts the signal.
> - It separates "may you train on this?" from "may you crawl this?". Visibility in AI answers depends on the second.
> - A wildcard group (`User-Agent: *`) does not count as this signal. You must name the token.
> - Adoption is uneven today. Keep GPTBot, ClaudeBot and Google-Extended rules until each provider confirms.

## What does the signal look like?

The draft's own example:

```
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
```

| Line | Meaning under this token |
|---|---|
| `User-Agent: __usage_ai-training` | The rules below are about AI training, not about a crawler |
| `Disallow: /` | Preference against AI training for the whole site |
| `Allow: /press-kit/` | Preference in favour of AI training for that folder |

Paths match as they do elsewhere in robots.txt, with the longest match winning. The two underscores at the start are part of the name.

## What counts as "AI training"?

The draft uses the working group's definition: using content to modify the learned parameters of a generative AI model. It does not address search, answering a user's question from retrieved pages, or fetching a page because a user asked. Those are outside the signal.

That is the point for AI visibility. Being cited in an answer depends on search crawlers reaching your pages, and on fetches made when a user asks, which this signal does not touch.

## What does it not do?

| It does not | What to use instead |
|---|---|
| Stop any crawler from fetching pages | A group that names the crawler, such as `User-agent: GPTBot` |
| Apply if you only have `User-Agent: *` rules | Add a group that names `__usage_ai-training` |
| Reach providers that have not adopted it | Their own crawler tokens |
| Remove content already collected | Nothing in this draft; it applies to the robots.txt in effect when a page is fetched |
| Enforce anything | It expresses a preference; the draft notes private agreements can override it |

If no group names the token, you have expressed no preference either way.

## Who honors it, and from when?

| Provider | Position | Source |
|---|---|---|
| Anthropic | Recognises the signal as of 30 October 2026, rolling out across its services; a rule naming ClaudeBot takes priority over the general signal | [Anthropic Help Center](https://support.claude.com/en/articles/8896518) |
| Google | Says it intends to support the signal in early 2027 | [Google crawler documentation](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) |
| Microsoft (Bing, Copilot) | Will follow the signal from 1 February 2027, as it crawls pages. A setting on a single page overrides robots.txt | [Microsoft Bing](https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/) |
| OpenAI | Announced on 30 October 2026 that it will honor the signal, reported as "in the coming months"; says existing crawler controls stay | Check [OpenAI's crawler page](https://platform.openai.com/docs/bots) |

We read the draft and the Anthropic, Google and Microsoft pages ourselves. The OpenAI row rests on its announcement as reported; confirm the date before you rely on it. Bing has also added a separate setting for AI answers, covered in [Bing AI settings: stay in Copilot answers](https://mentionx.ai/learn/technical/bing-ai-grounding-training-settings).

## Does opting out of training cost you AI visibility?

Under the signal as written, no. But providers differ in how they connect training to other uses, so check each one you care about:

- **Anthropic** runs separate bots: ClaudeBot for training, Claude-SearchBot and Claude-User for search and user requests. Opting out of training leaves the other two alone. See [ClaudeBot and Claude-SearchBot](https://mentionx.ai/learn/technical/claudebot-and-claude-searchbot).
- **Google** controls training and Gemini app grounding through Google-Extended, which is separate from Google Search, AI Overviews and AI Mode. See [Google-Extended controls](https://mentionx.ai/learn/technical/google-extended-controls).
- **OpenAI** separates GPTBot (training) from OAI-SearchBot (ChatGPT search).
- **Meta** separates Meta-ExternalAgent (training) from Meta-WebIndexer (Meta AI search). See [how to rank in Meta AI](https://mentionx.ai/learn/engines/meta-ai-how-to-rank).

*Our view:* for a brand that wants to be recommended by AI tools, the safe pattern is "no to training if you wish, yes to search and user requests". Blocking everything with "AI" in the name is the mistake that costs visibility.

## What should your robots.txt say today?

If you want to opt out of training and stay visible in AI answers:

```
# Shared training preference
User-Agent: __usage_ai-training
Disallow: /

# Keep provider specific training rules until each adopts the shared signal
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: meta-externalagent
Disallow: /

# Search and user-request agents stay allowed
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: meta-webindexer
Allow: /
```

Note the Google-Extended line: Google says that token also governs grounding in the Gemini app, so disallowing it has an effect beyond training. Decide that one deliberately.

If you are happy for your content to be used in training, you need no new lines. Saying nothing expresses no preference.

## How do you check it is in place?

1. Open `yourdomain.com/robots.txt` in a browser and find the group.
2. Confirm it is spelled exactly, with two leading underscores.
3. Confirm the agents used for search and for user questions are still allowed.
4. Repeat for each subdomain; robots.txt applies per host.
5. Put a date in your calendar to check provider support again each quarter and remove duplicate rules once they are no longer needed.

## Who should decide?

This is a business and legal choice, not a technical one. Marketing should bring one fact to that conversation: visibility in AI answers runs through the agents used for search and for user questions, and the training preference can be set without touching them. More in [blocking and allowing AI crawlers](https://mentionx.ai/learn/governance/blocking-allowing-ai-crawlers-copyright-licensing).

## Checklist

1. Decide your training preference with legal.
2. Add the `__usage_ai-training` group if you are opting out.
3. Keep provider specific training rules for now.
4. Confirm the agents for search and user questions are allowed.
5. Check every subdomain.
6. Review provider support each quarter.

## Related

- [How to configure robots.txt for AI crawlers](https://mentionx.ai/learn/technical/configure-robots-txt-ai-crawlers)
- [List of AI crawlers and user agents](https://mentionx.ai/learn/technical/list-ai-crawlers-user-agents)
- [ClaudeBot and Claude-SearchBot](https://mentionx.ai/learn/technical/claudebot-and-claude-searchbot)
- [Blocking and allowing AI crawlers: copyright and licensing](https://mentionx.ai/learn/governance/blocking-allowing-ai-crawlers-copyright-licensing)

## Sources

- IETF, [Machine-Readable Training Preference Signal](https://www.ietf.org/archive/id/draft-kelley-aipref-training-preference-signal-00.html) (Internet-Draft, 30 October 2026)
- Anthropic Help Center, [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518)
- Google, [Google's common crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)

## Frequently asked questions

### What is the AI training opt out line in robots.txt?

It is a short block you add to robots.txt that tells AI companies whether they may use your content to train their models. It was proposed on 30 October 2026 and uses the name __usage_ai-training.

### Does the AI training opt out block crawlers?

No. The draft says the token is not a crawling directive. It states a preference about use, and crawlers still follow the rules written for their own names.

### Will opting out of AI training remove my site from AI search answers?

The signal as defined covers training only, not search or answering questions. Check each provider, though. Google ties its training control to grounding in the Gemini app.

### Which AI companies honor the signal today?

Anthropic says it has honored it since 30 October 2026. Google says it intends to support it in early 2027. Microsoft and OpenAI have announced support with later start dates.

### Should I remove my GPTBot and ClaudeBot rules now?

No. Keep them. Most providers have not switched the shared signal on yet, and a rule that names a specific crawler still applies.

---

Written and reviewed by Dipuck Jones for MentionX Academy. The HTML version of this page: https://mentionx.ai/learn/technical/ai-training-opt-out-robots-txt-signal
