GPTBot User Agent: Allow or Block?

GPTBot is OpenAI's crawler for collecting content that may be used to train its models. Allow it if you want your content to inform future models; block it if you don't want that use. Either way, GPTBot is separate from OAI-SearchBot, which surfaces sites in ChatGPT search. Most brands that want AI visibility should allow OAI-SearchBot whatever they decide about GPTBot.

On this page

• What are OpenAI's crawlers?
• What does blocking GPTBot actually do?
• From MentionX data: how much answers rely on memory
• How should you decide? Four questions
• What robots.txt rules match each choice?
• What about other companies' training crawlers?
• How do you document and communicate the decision?
• How do you measure the effect of your decision?
• How do you verify it worked?
• Worked example: Ledgerline decides
• What are the common mistakes?
• Common questions
• Sources

Updated 31 October 2026. Anything marked "Our view" is MentionX's opinion, not OpenAI documentation. The worked example uses a fictional company. Check OpenAI's live crawler page before changing robots.txt.

What are OpenAI's crawlers?

OpenAI lists its user agents and what each does on its crawler page (OpenAI). In summary:

OpenAI also publishes IP ranges for its crawlers so you can verify traffic is genuine. Always check the live page; agents and wording change.

What does blocking GPTBot actually do?

It does:

• Signal that OpenAI shouldn't use your future crawled content for training.
• Reduce crawl load from that agent.

It doesn't:

• Remove you from ChatGPT search (that's OAI-SearchBot).
• Delete what was already collected or retrain existing models.
• Stop other companies' training crawlers (each has its own agent).
• Stop people pasting your content into ChatGPT.

From MentionX data: how much answers rely on memory

The case for allowing training crawlers rests on one idea: some answers come from what a model already knows, not from a live search. Our data suggests that's common.

Method. We ran read-only aggregate queries over completed MentionX runs from 22 August to 29 October 2026 (UTC): 2,154 runs across 18 anonymised projects, mostly B2B software. Six engines were called through their APIs with web search or grounding switched on where available. "Mentioned" means our extraction found the tracked brand in the answer text; "cited about the brand" means at least one returned citation was linked by our extraction to the brand, on its own site or elsewhere.

Even with search on, almost a third of OpenAI answers in this sample had no citations. In those answers, whatever the model said about a brand came from somewhere other than a page it showed the user, most likely its training.

Limits. A convenience sample through APIs, not the consumer ChatGPT app. A missing citation doesn't prove an answer came only from training; an engine may search and still show no links. We can't see which crawled pages went into any model.

Our view: if your brand is often named in uncited answers, training data matters to you, and that's a reason to think twice before blocking GPTBot on public marketing pages. If you sell content, the calculation is different.

How should you decide? Four questions

1. Is your content your product? Publishers, course creators, data providers and paid research firms often block training crawlers because their content is what they sell.
2. Do you want to be described well from memory? If buyers ask ChatGPT about your category without search, the model relies on what it learned. Allowing training crawlers is one input to being represented, though no one can guarantee how training data is used.
3. Are there legal or contractual limits? Licensed content, customer data, or regulated material may require blocking.
4. What's your policy for other AI companies? Make one decision and apply it to GPTBot, ClaudeBot, Google-Extended and others consistently.

These are common patterns, not rules. Your legal team has the final say.

What robots.txt rules match each choice?

Allow search and training:

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

Allow search, block training:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

Allow training on public content, block a private section for all agents:

User-agent: *
Disallow: /client-portal/

User-agent: GPTBot
Allow: /
Disallow: /client-portal/

Note the last example. A crawler follows the most specific group that matches its name, and ignores * when it has its own group (RFC 9309). If you add a GPTBot group, repeat any disallows you still want applied.

What about other companies' training crawlers?

GPTBot is one of several. If your reason for blocking is "we don't want our content used to train AI models", apply it consistently:

Sources: OpenAI, Anthropic, Google. Perplexity documents PerplexityBot as a search crawler (Perplexity); check its docs for its current position on training.

A policy that blocks GPTBot but leaves every other training crawler open is usually an accident, not a strategy.

How do you document and communicate the decision?

Write a short AI crawler policy, one page, and keep it next to robots.txt in your repo:

1. Decision: which agents are allowed or blocked, and for which paths.
2. Reason: for example "public marketing content; we want accurate representation in AI answers" or "licensed research; training use not permitted".
3. Owner: who approves changes.
4. Review date: at least every six months, and whenever a vendor adds a new agent.
5. How to verify: the log query your team uses.

Share it with content, legal and sales. Sales teams get asked "do you let AI train on your site?" more often than you might expect.

How do you measure the effect of your decision?

Changing a GPTBot rule won't show up in AI answers quickly, if at all, because model training happens on its own schedule. Measure what you can and be honest about what you can't.

What you can see straight away:

• Crawl behaviour. GPTBot requests to blocked paths should stop within days. Requests from OAI-SearchBot should carry on as before.
• Server load. If load was the reason for blocking, compare bandwidth and request counts for the agent before and after.

What you can track over months:

• Uncited answers. Keep a fixed prompt set and log separately the answers that cite nothing. Note whether they name you and whether their description is current. That's where training data shows.
• Model releases. When OpenAI releases a new model, re-run your prompts soon after and compare the uncited answers with the previous run.

What you can't see: which pages went into training, when, or how much weight they carried. Anyone who promises a measurable change from allowing GPTBot is guessing.

ChatGPT search is separate. If cited ChatGPT answers change after your robots.txt edit, look at OAI-SearchBot first. OpenAI says robots.txt changes can take about 24 hours to be reflected for search (OpenAI).

Our view: judge a GPTBot decision on principle (licensing, ownership, policy consistency), not on short-term visibility numbers. Search access is where the measurable gains are.

How do you verify it worked?

1. Fetch yoursite.com/robots.txt and confirm the live file has your rules (CDNs sometimes cache old versions).
2. In server logs, filter for GPTBot. After a block, requests should stop fetching disallowed paths.
3. Cross-check IPs against OpenAI's published ranges (OpenAI) to spot fake bots using the name.
4. Check your firewall isn't blocking OAI-SearchBot while you meant only to block GPTBot.

Worked example: Ledgerline decides

Ledgerline is a fictional accounts-payable tool for mid-sized companies. It has three kinds of content: a marketing site, public product docs, and a paid research library of AP benchmarks that customers buy.

The team works through the four questions:

1. Is the content the product? The research library is. The marketing site and docs aren't.
2. Do we want to be described well from memory? Yes. Their prompt tracking shows Ledgerline is often named in ChatGPT answers with no citations, and those answers use an old product description.
3. Legal limits? The research library includes licensed third-party data. Training use isn't allowed under that licence.
4. Other AI companies? The same rules should apply to ClaudeBot and Google-Extended.

The decision:

They write the rules with the research path disallowed in every named group, record the decision in a one-page policy with the legal owner and a six-month review date, and confirm in logs that GPTBot stops requesting /research/.

They also update the product description on their homepage, About page and main profiles, so whatever future training crawls collect is current. Illustrative: this shows the method, not a measured result.

What are the common mistakes?

• Blocking GPTBot and expecting to vanish from ChatGPT. Different agents.
• Adding a named group and forgetting it overrides ``.** Private paths become crawlable to that agent.
• Copying user agent lists from old blog posts. Use the vendor's documentation.
• Making the decision in SEO alone. Training use is a content-licensing question; involve legal and leadership.
• Blocking at the firewall by name only. A firewall rule matching "GPT" in the user agent can catch OAI-SearchBot and ChatGPT-User too. Match exact agent names and test each one.
• Forgetting subdomains. Docs and help centres on separate hosts need their own robots.txt with the same policy.
• Never revisiting the decision. Vendors add agents and change what they do with them. Put a review date in the policy and keep it.

• Treating robots.txt as security. It's advisory and public. Protect private content with authentication.

Common questions

Will blocking GPTBot hurt my Google rankings? No. GPTBot is OpenAI's crawler; Google Search uses Googlebot (Google).

Can I allow GPTBot on some pages only? Yes. Use Allow and Disallow lines for specific paths in the GPTBot group. Within a group, the most specific matching rule wins (RFC 9309).

Does GPTBot obey crawl-delay? Crawl-delay isn't part of RFC 9309. If GPTBot's load is a problem, rate-limit it at your server or CDN rather than relying on that line.

Should small companies bother with any of this? Yes, but it takes ten minutes. Most small B2B and ecommerce sites allow both crawlers and never think about it again. The one thing worth checking is that no CDN or hosting setting is quietly blocking OpenAI's search crawler.

How do I know a visit is really GPTBot? Check the IP against the ranges OpenAI publishes on its crawler page (OpenAI). Anything claiming to be GPTBot from another IP is an impostor.

If I allow GPTBot today, will the next model know about my latest product? Not necessarily. You can't see what goes into training or when a model's knowledge was gathered. Allowing the crawler only makes your content available to be collected.

Sources

• OpenAI, Overview of OpenAI crawlers
• IETF, RFC 9309: Robots Exclusion Protocol
• Google Search Central, Introduction to robots.txt; Google's common crawlers
• Anthropic, Crawler FAQ; Perplexity, Perplexity crawlers