Glossary · Beginner
What Is an AI Crawler? Definition and Example
An AI crawler is an automated bot that fetches web pages on behalf of an AI company, for training models, building an AI search index, or retrieving a page when a user asks a question. Each major company names its crawlers so site owners can manage them in robots.txt, and the purposes differ, so they should be managed separately.
The three kinds of AI crawler
| Purpose | Examples | Effect of blocking |
|---|---|---|
| Training | GPTBot, ClaudeBot; Google-Extended (a control token, not a crawler) | Future content excluded from training |
| AI search index | OAI-SearchBot, PerplexityBot, Claude-SearchBot | Less likely to appear or be cited in that engine's answers |
| User-initiated fetch | ChatGPT-User, Perplexity-User, Claude-User | Engine can't fetch your page when a user asks |
Sources: OpenAI, Perplexity, Anthropic, Google.
Example robots.txt
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /Illustrative: opting out of OpenAI training while allowing search crawlers.
What about Google AI Overviews?
Google's AI features in Search use the normal Search crawling and index; Google says there are no additional requirements (Google). Blocking Googlebot removes you from Search entirely, while Google-Extended doesn't affect Search.
How do you manage AI crawlers well?
- List which crawlers reach your site from logs.
- Decide per purpose: training, search, user fetch.
- Write explicit robots.txt groups per agent.
- Check your CDN or WAF isn't blocking ones you've allowed.
- Verify IPs, because user agents can be spoofed.
Why it matters for GEO
Accidentally blocking an AI search crawler is one of the most common, and most fixable, causes of low AI visibility.
Related
Questions
Do all AI crawlers respect robots.txt?
Most automatic crawlers from major companies say they do. Some user-initiated fetchers, such as ChatGPT-User and Perplexity-User, may not apply robots.txt.
Should I block AI crawlers?
Decide per purpose. Blocking training crawlers is a different choice from blocking search crawlers that can cite you.
How do I see AI crawler visits?
Search server or CDN logs for user agent tokens and verify IPs against published lists.