List of AI Crawlers and User Agents (2026)

AI crawlers fall into three groups: training crawlers (such as GPTBot and ClaudeBot), AI search crawlers (such as OAI-SearchBot, PerplexityBot and Claude-SearchBot) and user-initiated fetchers (such as ChatGPT-User, Perplexity-User and Claude-User). Each company documents its own agents and robots.txt tokens. This page lists the main ones with links to each official source, checked on 30 October 2026.

On this page

• What are the main AI crawlers?
• How should you group them?
• Which search crawler feeds which answer engine?
• From MentionX data: how much each engine cites
• Which ones affect Google AI Overviews?
• What does a balanced robots.txt look like?
• How do training and search decisions interact?
• How do you verify that a bot is genuine?
• How do you find each agent in your logs?
• Which policy fits which kind of site?
• How often should you review this list?
• Worked example: Ledgerline audits its crawlers
• What mistakes are common?
• Common questions
• Checklist
• Where to go next
• Sources

Checked against each company's crawler documentation on 30 October 2026. Anything marked "Our view" is MentionX's opinion. The worked example uses a fictional company.

What are the main AI crawlers?

Sources: OpenAI, Google common crawlers, Google AI features, Anthropic, Perplexity, Bing.

How should you group them?

Think in three columns rather than a list of names:

1. Training: GPTBot, ClaudeBot, Google-Extended. Blocking signals your future content shouldn't be used to train models.
2. AI search and indexing: OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot, Bingbot. Blocking reduces your chance of appearing or being cited in those answers.
3. User-initiated fetch: ChatGPT-User, Perplexity-User, Claude-User. These act when a person asks a question; OpenAI and Perplexity note robots.txt may not apply.

Your decision for each column can differ. Many publishers allow search and user fetch, and decide on training separately.

Which search crawler feeds which answer engine?

When you decide on search crawlers, it helps to know which answers each one feeds. Based on the companies' documentation:

From MentionX data: how much each engine cites

Method. We ran read-only aggregate queries over completed MentionX runs from 22 August to 29 October 2026 (UTC): 2,154 runs across 18 anonymised projects, mostly B2B software. Engines were called through their APIs with web search or grounding switched on where available.

Engines that cite on almost every answer, such as Perplexity and AI Mode, depend most on live retrieval. For them, blocking the search crawler or index removes the main route to being cited.

Limits. A convenience sample; Perplexity's is small. API behaviour can differ from the consumer apps, and the answer times include our own network overhead.

Our view: if you only have time to check a few agents, check Googlebot, Bingbot, OAI-SearchBot and PerplexityBot first. Together they sit behind most of the AI answers your buyers are likely to see.

Which ones affect Google AI Overviews?

Only Googlebot. Google says its AI features use the same Search systems, with no additional requirements: a page must be indexed and eligible to show a snippet (Google). Google-Extended doesn't affect inclusion in Google Search and isn't a ranking signal (Google). To keep a page out of AI features specifically, Google points to standard controls such as nosnippet, which also affect regular results.

What does a balanced robots.txt look like?

Illustrative; adapt to your own policy.

# Training
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

# AI search
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

# Everyone else
User-agent: *
Disallow: /admin/

Remember that a crawler follows the most specific group that matches it, and ignores the * group if it has its own (Google). If you add a named group, repeat any rules from * you still want it to follow.

How do training and search decisions interact?

They're separate controls, but the effects overlap in practice. An assistant answers a question about your category from two places: what the model learned in training, and what it retrieves from the web when the answer is built. Blocking a training crawler only affects the first, and only for future models. Blocking a search crawler affects the second straight away.

This shows up in MentionX data. In unbranded answers, engines named about 1.44 brands per answer when they gave no citations, compared with 1.11 when they did (n = 682 and 1,002 answers). Answers without retrieval still name brands, from memory. So a brand that models already know can keep showing up for a while after blocking training, and a newer brand that blocks search loses its main way in.

Our view: a young brand gains most from allowing both. An established brand can block training with less short-term risk, as long as search crawlers stay allowed and the pages that describe it are accurate.

A practical sequence for any change:

1. Write down the policy for each group and who approved it.
2. Change robots.txt and the CDN in the same release, so they don't drift apart.
3. Note the date, and track mentions and citations by engine for the following eight weeks.
4. Compare against the same prompts before the change. Answers vary from run to run, so look at rates over many runs, not single answers.

How do you verify that a bot is genuine?

Check a sample of requests from each claimed bot monthly. Impersonation is common, and blocking by user agent alone can be bypassed by scrapers.

How do you find each agent in your logs?

Your server or CDN access logs record every request with its user agent string, IP address, path and status code. For each agent you care about, pull four numbers for the last 30 days: total requests, the share that returned 200, the share that returned 403 or 429, and the top paths requested.

A simple filter on an Apache or Nginx combined log is enough to start:

grep -E "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|PerplexityBot|Perplexity-User" access.log \
  | awk '{print $9}' | sort | uniq -c

Then check a sample of IP addresses for each agent against the company's published ranges. Requests that claim to be a known bot but come from other networks are impersonators. Treat them as ordinary scrapers, and don't count them in your crawler reporting.

What to look for:

• Agents you allow that only receive 403 or 429. Something upstream of robots.txt, usually a CDN bot rule or rate limit, is blocking them.
• Agents you block that still receive 200. Either your robots.txt group doesn't match the token, or the agent is user-initiated and doesn't follow robots.txt.
• Crawls concentrated on low-value paths, such as faceted search URLs or old campaign pages. Tidy these with robots.txt rules, canonical tags or redirects, so that crawlers spend their visits on the pages you want cited.

Which policy fits which kind of site?

Our view, as a starting point rather than a rule:

For a brand that wants to be recommended, the cost of blocking search crawlers is direct: the engine can't fetch your pages when it builds an answer. The cost of blocking training crawlers is indirect and slower, because it affects what future models know about you without a search.

How often should you review this list?

At least quarterly, and whenever an AI product launches a new feature. OpenAI's list, for example, now includes OAI-AdsBot for ChatGPT ads landing pages, and Google notes it intends to support the AIPREF training preference signal in early 2027 (Google). Subscribe to the companies' documentation updates where offered; Anthropic offers a notification form on its crawler article.

Worked example: Ledgerline audits its crawlers

Ledgerline is a fictional accounts-payable tool for mid-sized companies. Its SEO lead builds a one-page crawler register for the marketing site and the docs subdomain.

Two rows don't match policy. A CDN rule written to block "GPT" scrapers also caught OAI-SearchBot and GPTBot, so ChatGPT search couldn't fetch pages Ledgerline wanted cited. The fix is to replace the pattern rule with exact agent names plus IP verification against OpenAI's published lists.

The register now lives in the website repo and is reviewed each quarter. Illustrative: the rows show the method, not real logs.

What mistakes are common?

• Blocking "all AI bots" with a CDN toggle, which may include search crawlers you want.
• Blocking ChatGPT-User to "stop ChatGPT" while OAI-SearchBot, which controls search inclusion, stays allowed, or the reverse.
• Assuming Google-Extended controls AI Overviews. It doesn't.
• Forgetting subdomains. robots.txt applies per host; Anthropic explicitly reminds site owners to apply rules on every subdomain.
• Never checking logs. Rules mean little if a firewall returns 403 before the bot reads them.

• Copying a robots.txt template from another site without checking that its policy matches yours.
• Treating impersonated traffic as real. Unverified "GPTBot" hits can make it look as if a block isn't working.

Common questions

Is there one list of all AI crawlers? No official one. Each company publishes its own. Third-party lists go out of date quickly, so check the vendor pages linked in the table.

What about crawlers not on this list? Many smaller AI companies and data brokers run crawlers. If an unknown agent causes load or ignores robots.txt, block or rate-limit it at your CDN. Robots.txt only works for crawlers that choose to follow it.

Do these crawlers run JavaScript? Don't assume they do. Make sure important text, prices and links are in the HTML your server sends, not only added by scripts after the page loads (Google).

Does blocking a crawler remove content it already collected? No. Robots.txt only affects future requests. Content gathered earlier may still be in training data or an index. For removal, use each company's own process where one exists, or Google's removal tools for Search.

How long do changes take to apply? Most crawlers re-read robots.txt within about a day, and Google says it generally caches the file for up to 24 hours. Allow a few weeks before the change shows up in answers.

Should I block Common Crawl's CCBot? It's a separate decision. Common Crawl's archive is widely used to build training datasets, so many sites treat it like a training crawler.

Checklist

• Decide policy for training, search and user fetch separately
• Write named robots.txt groups for each agent you treat differently
• Repeat needed * rules inside named groups
• Check CDN/WAF bot settings match robots.txt
• Verify IPs for a sample of requests
• Review the list quarterly

Where to go next

• Configure robots.txt for AI crawlers
• OAI-SearchBot vs GPTBot vs ChatGPT-User
• Reading server logs for AI bot visits

Sources

• OpenAI, Overview of OpenAI crawlers
• Google, Google's common crawlers and AI features and your website
• Anthropic, Does Anthropic crawl data from the web?
• Perplexity, Perplexity crawlers
• Microsoft, Which crawlers does Bing use?
• Google, How Google interprets the robots.txt specification
• Google, Verifying Googlebot and other Google crawlers
• Microsoft, Copilot and Bing web search
• Google, JavaScript SEO basics