OAI-SearchBot vs GPTBot vs ChatGPT-User

OpenAI runs three main agents: OAI-SearchBot surfaces sites in ChatGPT search, GPTBot crawls content that may be used to train its models, and ChatGPT-User visits pages when a ChatGPT user's action needs it. OpenAI says the settings are independent, so you can, for example, stay visible in ChatGPT search while opting out of training (OpenAI). This guide explains each one and four common policies.

Key takeaways

• Search visibility in ChatGPT depends on OAI-SearchBot, not GPTBot.
• GPTBot is about training; disallowing it indicates content shouldn't be used to train foundation models.
• ChatGPT-User is user-initiated; robots.txt may not apply, and it doesn't decide search inclusion.
• Changes can take about 24 hours to reach OpenAI's search systems.

What does each agent do?

All details from OpenAI's crawler documentation.

Which policy fits you?

These are common patterns, not recommendations for any specific business. Decide with legal and content owners.

What do the robots.txt rules look like?

Visible in search, no training:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

Keep a private section out of everything:

User-agent: OAI-SearchBot
Disallow: /members/

User-agent: GPTBot
Disallow: /

Rules are matched to the most specific user-agent group, so if you've written a named group for OAI-SearchBot, it won't also follow your User-agent: * rules (Google's description of the standard, RFC 9309). Copy any shared disallows into each named group.

What about ChatGPT-User?

ChatGPT-User fetches a page when a person in ChatGPT, or a Custom GPT, needs it, for example to read a URL they shared. OpenAI says it's not used to crawl the web automatically, robots.txt rules may not apply, and it isn't used to decide whether content appears in search. If you want to manage search presence, use OAI-SearchBot.

If you see ChatGPT-User in logs, it often means a real user asked ChatGPT about your page. It can be a useful qualitative signal of interest.

How do you check what's happening?

1. Read robots.txt at your root and on each subdomain.
2. Search logs for OAI-SearchBot, GPTBot and ChatGPT-User. OpenAI notes it may add a robots.txt marker to the user agent when fetching that file.
3. Compare IPs with OpenAI's published JSON lists.
4. Check response codes. Repeated 403s mean a firewall is blocking regardless of robots.txt.
5. Test in ChatGPT. Ask questions your page answers with search on and see whether it's cited, allowing about 24 hours after robots.txt changes.

What about OAI-AdsBot?

OpenAI also lists OAI-AdsBot, which visits landing pages submitted as ads on ChatGPT to check them against policies and judge relevance. OpenAI says it only visits submitted pages and its data isn't used to train foundation models. It's irrelevant unless you run ChatGPT ads.

Worked example

Illustrative. A B2B publisher blocked "GPTBot" two years ago and assumed it was out of ChatGPT. Its marketing team now wants citations. Reviewing robots.txt shows no OAI-SearchBot group, so OAI-SearchBot follows the * group, which allows most content. Logs show OAI-SearchBot requests returning 403 from a CDN bot rule. The fix is in the CDN, not robots.txt. After allowing verified OpenAI IPs, the team waits a few days, then checks prompts in ChatGPT search.

How do you decide on GPTBot?

The GPTBot decision isn't really a marketing one. It's about whether you're comfortable with your content potentially being used in model training. Questions to settle with legal and content owners:

• Do licences or contracts restrict how your content may be reused?
• Is your content your product (journalism, research, courses), or marketing for a product?
• Would you rather negotiate content licensing than allow crawling by default?
• Are you consistent across companies? If you block GPTBot, should ClaudeBot and Google-Extended be treated the same way?

Document the decision and the date, so future teams don't reverse it by accident during a site migration.

How often should you revisit the decision?

Review your OpenAI rules at least quarterly and whenever OpenAI updates its crawler documentation. Business priorities change too: a publisher that blocked GPTBot for licensing reasons may later sign an agreement, and a brand that allowed everything may decide training access isn't worth it. Record each decision with its date and reason so the next person understands it.

Common mistakes

• Treating GPTBot and OAI-SearchBot as one switch.
• Blocking ChatGPT-User expecting it to remove search visibility.
• Forgetting robots.txt group precedence, so named groups lose * rules.
• Not checking the CDN.

Checklist

• [ ] Policy agreed for search and training separately
• [ ] Named groups for OAI-SearchBot and GPTBot
• [ ] Shared disallows repeated in each named group
• [ ] Logs show 200 responses for verified OpenAI IPs
• [ ] Spot-check in ChatGPT search after 24 hours

Where to go next

• List of AI crawlers and user agents
• OAI-SearchBot
• Configure robots.txt for AI crawlers

Sources

• OpenAI, Overview of OpenAI crawlers
• Google, How Google interprets the robots.txt specification
• IETF, RFC 9309: Robots Exclusion Protocol