PerplexityBot and Perplexity-User Explained

Perplexity uses two agents: PerplexityBot crawls and indexes sites so Perplexity can surface and link them in its answers, and Perplexity-User fetches pages when a user's question needs them. Perplexity recommends allowing PerplexityBot in robots.txt and permitting requests from its published IP ranges if you want to appear in results (Perplexity). Most visibility problems come from firewalls rather than robots.txt.

Key takeaways

• Allow PerplexityBot if you want to be cited in Perplexity.
• Perplexity says neither agent is used to train foundation models.
• Perplexity-User generally ignores robots.txt, as a user requested the fetch.
• Match user agent and IP range in firewall rules.

How do the two agents compare?

Source: Perplexity crawlers documentation.

What should robots.txt say?

To allow indexing everywhere except private areas:

User-agent: PerplexityBot
Allow: /
Disallow: /account/

To opt out of Perplexity's index:

User-agent: PerplexityBot
Disallow: /

Opting out of PerplexityBot doesn't stop Perplexity-User fetching a page a user asked about. If a page must never be fetched, protect it with authentication rather than robots.txt.

How do you configure your firewall?

Perplexity's documentation gives step-by-step guidance for two common WAFs.

Cloudflare:

1. Go to Security → Security rules and create a custom rule.
2. Match each bot's user agent with its own IP ranges, for example (http.user_agent contains "PerplexityBot" and ip.src in {...}).
3. Set the action to Skip for the protections you want bypassed.
4. Place the rule above any blocking or challenge rules.

Perplexity warns that Cloudflare's Bot Fight Mode can't be skipped with custom rules and may still challenge its bots.

AWS WAF:

1. Create an IP set per bot from the official JSON lists.
2. Create an Allow rule combining the IP set match with a User-Agent string match.
3. Give these rules higher priority (lower numbers) than blocking rules.

Perplexity recommends automating IP list updates, since ranges change.

How do you troubleshoot low Perplexity visibility?

How do you verify genuine traffic?

Compare request IPs with Perplexity's published JSON lists. A request claiming to be PerplexityBot from an IP outside the list isn't from Perplexity, and is safe to treat as you would any unknown scraper.

Worked example

Illustrative. An ecommerce site allows PerplexityBot in robots.txt but sees almost no Perplexity citations. Its CDN logs show PerplexityBot requests receiving managed challenges. The team adds a custom Skip rule matching user agent and IP ranges above its challenge rules, then disables Bot Fight Mode, which Perplexity notes can't be skipped. Within a week, logs show 200 responses, and the team starts monthly prompt checks to see whether citations follow.

Should you block Perplexity-User at the firewall?

Usually not, if you want visibility. Perplexity-User fetches pages for people actively asking about a topic, and blocking it means Perplexity can't read your page when a user points to it or needs it for an answer. If specific pages must stay private, put them behind a login. If you're worried about load, monitor request volume first; user-initiated fetches are driven by real questions rather than bulk crawling.

How should you read Perplexity-User visits?

Because Perplexity-User fetches pages for real questions, a spike can show interest in a page or topic. Treat it as qualitative: log which URLs it visits and compare with your prompt research.

What content tends to work in Perplexity?

Perplexity shows its sources next to the answer, so readers see exactly which pages it relied on. Being crawlable is the precondition; after that, pages that answer a specific question directly, with clear facts and dates, are easier to cite. There's no published list of Perplexity ranking factors, so test with your own prompts rather than relying on claims about what it "prefers".

A practical routine:

1. Pick 10 prompts your buyers ask.
2. Run them in Perplexity and log the cited domains.
3. Compare cited pages with yours: structure, specificity, freshness, depth.
4. Improve one page and recheck the same prompts in a few weeks.

How do you monitor ongoing access?

Set up a weekly log query, or a CDN dashboard filter, counting PerplexityBot requests and response codes. A sudden drop to zero, or a jump in 403s, usually follows a firewall or CDN configuration change. Catching it within a week is far cheaper than discovering months later that citations dried up.

Who should own Perplexity access settings?

Usually two people share it. Marketing or SEO decides the policy, and whoever manages the firewall or CDN implements it. Problems start when one changes settings without telling the other, for example a security team adding a bot rule that silently blocks PerplexityBot. Agree a short change process and log every rule change with its date.

Common mistakes

• Allowing PerplexityBot in robots.txt while a WAF blocks it.
• Hard-coding IPs once and never updating them.
• Relying on robots.txt to keep pages private from Perplexity-User.
• Confusing the two agents when analysing logs.

Checklist

• [ ] PerplexityBot allowed in robots.txt (if desired)
• [ ] WAF rule combining user agent and IP list, placed above blocks
• [ ] Bot Fight Mode reviewed (Cloudflare)
• [ ] IP lists updated automatically
• [ ] Logs show 200 responses for verified IPs

Where to go next

• PerplexityBot (glossary)
• List of AI crawlers and user agents
• Reading server logs for AI bot visits

Sources

• Perplexity, Perplexity crawlers
• Perplexity, PerplexityBot IP list