Glossary · Beginner
What Is OAI-SearchBot? Definition and Example
OAI-SearchBot is the web crawler OpenAI uses to surface websites in ChatGPT's search features. If you want your pages to appear as sources in ChatGPT search answers, OpenAI recommends allowing OAI-SearchBot in robots.txt and allowing requests from its published IP ranges (OpenAI). It is separate from GPTBot, which relates to model training.
What does OAI-SearchBot do?
It crawls pages so ChatGPT can find and link them when it searches the web for an answer. OpenAI states that sites opted out of OAI-SearchBot won't be shown in ChatGPT search answers, although they can still appear as navigational links (OpenAI).
How is it different from OpenAI's other agents?
| Agent | Purpose (per OpenAI) | robots.txt applies? |
|---|---|---|
| OAI-SearchBot | Surfacing sites in ChatGPT search | Yes |
| GPTBot | Crawling content that may be used to train foundation models | Yes |
| ChatGPT-User | Visiting pages for certain user actions | May not apply, as user-initiated |
| OAI-AdsBot | Checking landing pages submitted as ChatGPT ads | Visits submitted ad pages only |
Example robots.txt
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /Illustrative: a publisher that doesn't want its articles used for training, but does want ChatGPT search to cite them, might use exactly this. OpenAI notes that if both bots are allowed, it may use one crawl for both purposes to avoid duplicate crawling.
How do you check it's reaching your site?
Search your access logs for OAI-SearchBot. Because user agent strings can be spoofed, compare the request IPs with OpenAI's published list at openai.com/searchbot.json. OpenAI also notes it may add a robots.txt marker to the user agent when fetching that file, which helps if your logs don't record paths.
Why it matters for GEO
ChatGPT search can only cite pages it can reach. A blanket "block all AI bots" rule, or a firewall rule written for scrapers, can quietly remove you from those answers. Check robots.txt and your CDN's bot settings whenever ChatGPT visibility drops unexpectedly.
Related
Questions
Does blocking GPTBot block OAI-SearchBot?
No. OpenAI says each setting is independent, so you can block training (GPTBot) and still allow search (OAI-SearchBot).
What happens if I block OAI-SearchBot?
OpenAI says opted-out sites won't be shown in ChatGPT search answers, though they can still appear as navigational links.
How fast do robots.txt changes take effect?
OpenAI says it can take about 24 hours for its search systems to adjust.