Glossary · Beginner
What Is PerplexityBot? Definition and Example
PerplexityBot is the crawler Perplexity uses to surface and link websites in its search results. Perplexity says it is not used to crawl content for AI foundation models, and recommends allowing it in robots.txt, plus its published IP ranges, if you want to appear in results (Perplexity).
What does PerplexityBot do?
Perplexity answers questions with cited sources. PerplexityBot builds the index those sources come from. If it can't crawl your pages, they're less likely to be among the links Perplexity shows.
PerplexityBot vs Perplexity-User
| PerplexityBot | Perplexity-User | |
|---|---|---|
| Purpose | Surface and link sites in search results | Visit pages to answer a specific user question |
| Triggered by | Automatic crawling | A user's request |
| Respects robots.txt | Yes | Perplexity says it generally ignores robots.txt, since the user requested the fetch |
| Used for training? | No, per Perplexity | No, per Perplexity |
| IP list | perplexity.ai/perplexitybot.json | perplexity.ai/perplexity-user.json |
Example robots.txt
User-agent: PerplexityBot
Allow: /What about firewalls?
Perplexity's documentation includes WAF guidance: match the user agent and the bot's own published IP ranges together, and place the allow rule above blocking rules. It warns that Cloudflare's Bot Fight Mode can't be skipped with custom rules and may still challenge its bots (Perplexity).
Illustrative: a site allowing PerplexityBot in robots.txt but running aggressive bot protection at the CDN may still be invisible to Perplexity. Checking logs for successful (200) responses to PerplexityBot is the real test.
Why it matters for GEO
Perplexity shows citations prominently, so being crawlable is a precondition for one of the most visible kinds of AI referral. Review robots.txt, WAF rules and IP allowlists together.
Common mistakes
- Assuming robots.txt alone is enough when a CDN or WAF challenges bots before they reach the file.
- Confusing Perplexity-User with PerplexityBot when reading logs.
Related
Questions
Does PerplexityBot crawl content for training?
Perplexity says PerplexityBot is not used to crawl content for AI foundation models.
What's Perplexity-User?
A separate agent that fetches pages when a user asks a question. Perplexity says it generally ignores robots.txt because the user requested the fetch.
How long until robots.txt changes apply?
Perplexity says it may take up to 24 hours for its systems to reflect changes.