# PerplexityBot, robots.txt and WAF settings for marketers

> Let Perplexity reach your pages: PerplexityBot vs Perplexity-User, robots.txt examples, Cloudflare and AWS WAF rules, and how to check logs.

Source: https://mentionx.ai/learn/perplexity/perplexity-perplexitybot-robots-waf  
Author: Dipuck Jones, founder of MentionX  
Updated: 5 November 2026  
Level: Practitioner

**If Perplexity can't reach a page, it can't cite it. Perplexity uses two user agents. PerplexityBot crawls for search results and follows robots.txt. Perplexity-User fetches pages when someone asks a question and generally ignores robots.txt, because a person asked for it. To be cited, allow PerplexityBot in robots.txt and make sure your firewall or CDN doesn't block either agent from Perplexity's published IP ranges. Then check your server logs to confirm it works.**

> **Key takeaways**
>
> - PerplexityBot = search crawler, follows robots.txt, and isn't used for training foundation models (per Perplexity).
> - Perplexity-User = user-requested fetches, which generally ignore robots.txt.
> - Firewalls and bot protection are the most common hidden blocker. Allow by user agent and IP together.
> - Changes can take up to 24 hours. Verify with server logs, not assumptions.

*Checked 5 November 2026 (CET) against the Perplexity Help Center, Perplexity blog, Perplexity API docs and the outlets cited in Sources. Perplexity renames and moves features often, so confirm labels and limits in your own account. "Our view" marks MentionX opinion. Ledgerline is a fictional company used for illustration.*

## What are Perplexity's crawlers?

From Perplexity's crawler documentation (checked November 2026):

| User agent | Purpose | robots.txt | Example UA string | IP list |
|---|---|---|---|---|
| PerplexityBot | Surface and link websites in Perplexity search results | Follows it | `Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)` | perplexity.ai/perplexitybot.json |
| Perplexity-User | Fetch a page to answer a user's question | Generally ignores it, as a user-requested fetch | `Mozilla/5.0 (compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)` | perplexity.ai/perplexity-user.json |

The version numbers in the strings may change. Match on the name, not the full string. For the wider bot landscape, see the [list of AI crawlers and user agents](https://mentionx.ai/learn/technical/list-ai-crawlers-user-agents) and the [PerplexityBot glossary entry](https://mentionx.ai/glossary/perplexitybot).

## What should robots.txt say?

**Allow Perplexity (recommended if you want citations):**

```
User-agent: PerplexityBot
Allow: /

User-agent: *
Disallow: /admin/
Disallow: /checkout/
```

**Allow, but keep private areas out:**

```
User-agent: PerplexityBot
Disallow: /internal/
Disallow: /customer-portal/
Allow: /
```

**Block Perplexity search crawling (if your business decides to opt out):**

```
User-agent: PerplexityBot
Disallow: /
```

Notes:

- Blocking PerplexityBot reduces your chances of appearing in Perplexity search results. Make this a deliberate business decision, not an accident.
- Robots.txt doesn't reliably stop Perplexity-User. Use authentication or paywalls for content that must stay private ([paywalls and logins](https://mentionx.ai/learn/technical/paywalls-logins-ai-access)).
- Check for broad rules like `User-agent: *` with `Disallow: /` left over from staging.

More on robots.txt patterns: [configure robots.txt for AI crawlers](https://mentionx.ai/learn/technical/configure-robots-txt-ai-crawlers).

## How do you allow Perplexity through Cloudflare?

Per Perplexity's documentation:

1. In the Cloudflare dashboard, go to **Security → Security rules**.
2. Select **Create rule → Custom rules**.
3. Select **Edit expression** and match each bot's user agent to its own IP ranges:
```
(http.user_agent contains "PerplexityBot" and ip.src in {<PerplexityBot IP ranges>})
or (http.user_agent contains "Perplexity-User" and ip.src in {<Perplexity-User IP ranges>})
```
4. Set the action to **Skip** and choose what to bypass, such as remaining custom rules, managed rules, rate limiting rules or Super Bot Fight Mode rules.
5. Place the rule **above** any rules that block or challenge traffic.

Points to know:

- On paid plans you can store ranges in IP lists (for example `ip.src in $perplexitybot`) and update the list instead of the rule. Free plans allow one custom list, which isn't enough for both bots.
- **Bot Fight Mode can't be skipped** with custom rules. If it's on, it may still challenge Perplexity.
- Cloudflare also has its own AI crawler controls. Check that they match your robots.txt intent ([Cloudflare AI bot blocking check](https://mentionx.ai/learn/technical/cloudflare-ai-bot-blocking-check-youre)).

## How do you allow Perplexity through AWS WAF?

1. Create an IP set for PerplexityBot and one for Perplexity-User from the JSON endpoints.
2. For each bot, create a rule with an **AND** statement: IP set match, plus a User-Agent header string match containing the bot name.
3. Set the action to **Allow**.
4. Give these rules **lower priority numbers** than your blocking rules so they're evaluated first.

For other WAFs and CDNs, the principle is the same: allow when both the user agent and the published IP match. Never allow by user agent alone, because user agents are trivial to fake.

> **Pro move:** automate the IP list refresh. Perplexity says its published ranges are updated regularly. A small scheduled job (n8n or a cron script) that fetches both JSON files and updates your WAF list prevents silent blocks months later.

## How do you check it's working?

1. **Robots.txt test.** Fetch your robots.txt and read the rules for PerplexityBot and `*`.
2. **Server or CDN logs.** Filter for "PerplexityBot" and "Perplexity-User" user agents. Check status codes: 200 is good, while 403, 429 or challenge pages mean you're blocking ([reading server logs for AI bots](https://mentionx.ai/learn/technical/reading-server-logs-ai-bot-visits)).
3. **Verify the IPs** against the JSON lists, since fake bots use real names.
4. **Live test.** Ask Perplexity a question that only a new page answers, or paste the URL and ask for a summary. If it can't access the page, check the WAF.
5. **Wait up to 24 hours** after robots.txt changes before retesting.

## What was the Cloudflare dispute about?

On 4 August 2025 Cloudflare published research accusing Perplexity of "stealth" crawling, saying it used undeclared user agents and IPs to get around blocks. Perplexity disputed this in its post "Agents or Bots? Making Sense of AI on the Open Web". It argued the traffic was user-initiated and that Cloudflare had confused it with a third-party browser service's traffic. Cloudflare said it removed Perplexity from its verified-bot list and added blocking rules. The Verge and Ars Technica covered both sides.

*Our view:* for marketers who want to be cited, the practical point is simple. Allow the declared agents from their published IPs, block what you need to block deliberately, and monitor logs. If you want to keep content out of AI systems, robots.txt alone isn't enough. Use access controls.

## Worked example: Ledgerline

*Ledgerline is a fictional accounts-payable software company. Details are illustrative.*

Ledgerline's product pages were rarely cited, even after content fixes. The SEO lead checked access.

| Check | Finding | Fix |
|---|---|---|
| robots.txt | PerplexityBot allowed | None needed |
| CDN logs | Perplexity-User requests to /pricing returned 403 | Cloudflare skip rule using IP lists |
| Bot Fight Mode | On | Switched off for the marketing site after a security review, with rate limiting kept |
| Docs subdomain | Separate robots.txt blocked all bots | Allowed PerplexityBot on public docs |
| IP list refresh | Manual | Weekly n8n job |

### Walkthrough

1. Pulled 30 days of CDN logs filtered by Perplexity user agents.
2. Found 403s on the pricing and comparison pages.
3. Created the Cloudflare skip rule with IP lists, above the block rules.
4. Reviewed Bot Fight Mode with security and agreed a change.
5. Fixed the docs subdomain robots.txt.
6. Retested after 24 hours. The logs showed 200s, and Perplexity could summarise the pricing page.

## Common mistakes

- **Allowing by user agent only.** Always pair it with the IP lists.
- **Forgetting subdomains.** Each one has its own robots.txt.
- **Bot protection that silently challenges.** Check status codes in the logs.
- **Expecting robots.txt to block Perplexity-User.** Use access controls instead.

## Try it yourself

Open your robots.txt and your CDN or server logs. Search the logs for both Perplexity user agents over the last 30 days and list the status codes by URL path. If you find 403s or challenges on public pages, draft the WAF rule for your security team.

## Quick quiz

**1. Which Perplexity agent follows robots.txt?**

*Answer:* PerplexityBot. Perplexity-User generally ignores it for user-requested fetches.

**2. Why match on IP as well as user agent?**

*Answer:* Because user agents are easy to fake.

**3. What can't Cloudflare custom rules skip?**

*Answer:* Bot Fight Mode.

## What should you check before the next lesson?

- robots.txt allows PerplexityBot on public pages, including subdomains.
- Logs show 200 responses to both agents on key pages.
- IP lists refresh automatically.

Previous: [How Perplexity chooses sources](https://mentionx.ai/learn/perplexity/perplexity-how-sources-are-chosen). Next: [Content formats that get cited](https://mentionx.ai/learn/perplexity/perplexity-content-formats-get-cited). Or return to the [Perplexity for Marketers track](https://mentionx.ai/learn/perplexity/perplexity-for-marketers). Related: [PerplexityBot and Perplexity-User](https://mentionx.ai/learn/technical/perplexitybot-perplexity-user), [Perplexity SEO audit](https://mentionx.ai/learn/engines/perplexity/seo-audit), [JavaScript rendering for AI crawlers](https://mentionx.ai/learn/technical/javascript-rendering-ai-crawlers).

## Common questions

**What is the difference between PerplexityBot and Perplexity-User?** PerplexityBot crawls to surface and link sites in Perplexity search results and follows robots.txt. Perplexity-User fetches pages when a user asks a question and, as a user-requested fetch, generally ignores robots.txt.

**Does PerplexityBot train AI models?** Perplexity says PerplexityBot isn't used to crawl content for AI foundation models. It's used for search results.

**How long do robots.txt changes take to apply?** Perplexity says it may take up to 24 hours for its systems to reflect robots.txt changes.

## Sources

- Perplexity API docs, [Perplexity Crawlers](https://docs.perplexity.ai/docs/resources/perplexity-crawlers)
- Perplexity blog, [Agents or Bots? Making Sense of AI on the Open Web](https://www.perplexity.ai/hub/blog/agents-or-bots-making-sense-of-ai-on-the-open-web)
- Cloudflare blog, [Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives](https://blog.cloudflare.com/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives/)
- The Verge, [Cloudflare says Perplexity's AI bots are 'stealth crawling' blocked sites](https://www.theverge.com/news/718319/perplexity-stealth-crawling-cloudflare-ai-bots-report)
- Ars Technica, [AI site Perplexity uses "stealth tactics" to flout no-crawl edicts, Cloudflare says](https://arstechnica.com/information-technology/2025/08/ai-site-perplexity-uses-stealth-tactics-to-flout-no-crawl-edicts-cloudflare-says/)
- Cloudflare docs, [Custom rules](https://developers.cloudflare.com/waf/custom-rules/)
- All accessed 5 Nov 2026.

## Frequently asked questions

### What is the difference between PerplexityBot and Perplexity-User?

PerplexityBot crawls to surface and link sites in Perplexity search results and follows robots.txt. Perplexity-User fetches pages when a user asks a question and, as a user-requested fetch, generally ignores robots.txt.

### Does PerplexityBot train AI models?

Perplexity says PerplexityBot is not used to crawl content for AI foundation models. It is used for search results.

### How long do robots.txt changes take to apply?

Perplexity says it may take up to 24 hours for its systems to reflect robots.txt changes.

---

Written and reviewed by Dipuck Jones for MentionX Learn. The HTML version of this page: https://mentionx.ai/learn/perplexity/perplexity-perplexitybot-robots-waf