Glossary · Beginner
What Is GPTBot? Definition and Example
GPTBot is OpenAI's web crawler for collecting publicly available content that may be used to train OpenAI's generative AI foundation models. You control it in robots.txt with the user agent GPTBot. It's one of several OpenAI agents; ChatGPT's search features use a different one, OAI-SearchBot (OpenAI docs).
What are OpenAI's crawlers?
| User agent | Purpose (per OpenAI) |
|---|---|
| GPTBot | Training generative AI foundation models |
| OAI-SearchBot | Surfacing websites in ChatGPT search features |
| ChatGPT-User | Actions started by users, such as visiting a page they ask about |
Check OpenAI's page for current details; it changes (OpenAI docs).
Example robots.txt
To allow search and block training:
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /Each group applies only to the agent it names. Under Google's documented rules, a crawler follows the most specific group that matches it (Google robots.txt spec). OpenAI says its crawlers respect robots.txt.
Should you block it?
It depends on how you feel about your content being used for training. Blocking GPTBot doesn't stop OAI-SearchBot. What actually hurts visibility is blocking OAI-SearchBot by accident, often with a blanket User-agent: * disallow.
How do you check whether GPTBot visits?
Search your server or CDN logs for GPTBot in the user-agent string, and verify the IPs against OpenAI's published ranges.
What are the common mistakes?
- Assuming GPTBot is the bot behind ChatGPT search. It isn't; OAI-SearchBot is.
- Copying an "AI blocklist" from a forum without checking what each agent does.
- Trusting the user-agent string alone. It can be spoofed, so verify the IPs.
- Forgetting that robots.txt changes take time to be picked up.
Related
Questions
Does blocking GPTBot remove me from ChatGPT search?
No. ChatGPT search uses OAI-SearchBot. OpenAI documents the agents separately, and each follows its own robots.txt rules.
How do I block GPTBot?
Add a "User-agent GPTBot" group with "Disallow /" to your robots.txt file.
Should I block GPTBot?
It is a business decision about training use. Many brands that want AI visibility allow OAI-SearchBot whatever they decide about GPTBot.