Cloudflare AI Bot Blocking: Check You're Not Invisible

If your site uses Cloudflare, check Security Settings > Configure AI bot policies: Cloudflare lets you block Search, Agent and Training bots separately, and since 15 September 2026 new domains block Training and Agent bots on pages with ads by default. Blocking Agent or Search bots can stop AI assistants fetching your pages for live answers. Then check your served robots.txt, because Cloudflare can prepend managed directives to it (Cloudflare, Cloudflare).

Facts are from Cloudflare documentation accessed 30 Oct 2026; settings and names change often, so confirm in your dashboard. Points marked "Our view" are interpretation.

On this page

• Why does this matter for AI visibility?
• How did Cloudflare's defaults change?
• What do Search, Agent and Training mean?
• What does managed robots.txt do?
• What is AI Crawl Control?
• How do you check your site in five minutes?
• What settings make sense for most brands?
• Who should make the decision?
• Checklist
• From MentionX data: why access matters for citations
• Worked example: Ledgerline finds a silent block
• How do you tell a block from other problems?
• What about robots.txt you already have?
• Which teams need to be involved?
• How often should you re-check?
• What do people get wrong when checking?
• What should you record after the check?
• Common questions
• Related
• Sources

Checked on 31 October 2026. "Our view" marks MentionX opinion. Ledgerline is a fictional company used for illustration.

Why does this matter for AI visibility?

AI answer engines need to fetch your pages to cite them. If your CDN blocks their crawlers or real-time fetchers, you may vanish from answers that would otherwise use your content, even with perfect content. Our view: this is one of the most common hidden causes of "we're not cited anywhere", and it's often set by someone in security or IT rather than marketing.

How did Cloudflare's defaults change?

• 1 July 2025: Cloudflare announced it was changing the default to block AI crawlers unless they pay creators for their content, calling it "Content Independence Day" (Cloudflare blog).
• 15 September 2026: for new domains, bots classified as Training or Agent are blocked on pages that display ads, and Search remains allowed. Mixed-purpose crawlers combining Search and Training are blocked by any configuration that blocks AI training, including the legacy "Block AI bots" option, which is being deprecated (Cloudflare).

Existing domains keep the settings they had unless someone changes them. That's why checking matters: you can't tell from the date your site launched what's on.

What do Search, Agent and Training mean?

Cloudflare's presets group AI bots by behaviour (Cloudflare):

Each setting has three options: Block on all pages, Block on pages with ads, or Allow (Cloudflare). Each blocking option covers verified bots in that category plus some unverified bots that behave similarly.

What does managed robots.txt do?

With Bot Preference Sync turned on and at least one policy set to block, Cloudflare prepends managed directives and a Content Signals Policy to your robots.txt. If your origin already serves a robots.txt with a 200 response, Cloudflare puts its managed content before yours in a single response (Cloudflare).

The content signals cover search, ai-input (such as retrieval and grounding for generative answers) and ai-train. Cloudflare's own text notes that search doesn't include AI-generated search summaries (Cloudflare).

Cloudflare also states that robots.txt compliance is voluntary and doesn't prevent access technically; to enforce restrictions it points to AI Crawl Control (Cloudflare).

What is AI Crawl Control?

AI Crawl Control, formerly AI Audit, is available on all plans. It shows which AI services access your content, lets you set allow or block rules per crawler, tracks robots.txt compliance and offers Pay Per Crawl in beta (Cloudflare).

How do you check your site in five minutes?

1. Dashboard: open Security Settings > Configure AI bot policies and note the Search, Agent and Training settings (Cloudflare).
2. AI Crawl Control: check which crawlers are blocked or allowed, and which have been making requests (Cloudflare).
3. Served robots.txt: fetch it and look for a "BEGIN Cloudflare Managed content" block:

bash curl -s https://example.com/robots.txt | head -60

1. Custom WAF rules: search for rules matching AI user agents or bot categories that someone added manually.
2. Server logs: confirm whether the crawlers you care about receive 200s or 403s. See Reading server logs for AI bot visits.
3. Vendor docs: match user agents against the vendors' own lists (OpenAI, Anthropic, Perplexity).

What settings make sense for most brands?

Our view, not Cloudflare guidance:

Remember that mixed-purpose crawlers are blocked whenever you block training (Cloudflare). If an engine uses one crawler for both, blocking training can also reduce search use. Decide that trade-off deliberately.

Who should make the decision?

Our view: settle it jointly between marketing, legal and security, write it down, and review it when Cloudflare changes defaults. The trade-offs are covered in Blocking or allowing AI crawlers: copyright and licensing, and robots.txt rules in Configure robots.txt for AI crawlers.

Checklist

• Note Search, Agent, Training settings
• Check AI Crawl Control per-crawler rules
• Inspect served robots.txt for managed block
• Review custom WAF rules
• Confirm crawler status codes in logs
• Record the decision and owner

From MentionX data: why access matters for citations

Method. Read-only aggregate queries over MentionX scans from 22 August to 29 October 2026 (UTC): 2,217 engine runs (2,154 completed) across 18 anonymised projects and six engines.

Engines can only cite pages they can fetch. In our sample, engines differed sharply in how often they showed sources at all:

For engines that cite often, a blocked crawler or fetcher removes a real chance of being linked. For engines that rarely cite, the effect shows up as less accurate descriptions rather than missing links. The cited pages we could classify break down like this:

Blog and article pages, documentation and pricing pages are the kinds of content most often behind a CDN with security rules. Our view: if any of those page types return errors to AI crawlers, check that first.

Limits. A convenience sample of mostly B2B software projects, run through engine APIs, which can behave differently from the consumer apps. Perplexity, Copilot and DeepSeek figures come from a single day (26 October 2026), so treat them as a snapshot, not a trend.

Worked example: Ledgerline finds a silent block

Ledgerline is a fictional accounts-payable tool. Figures are illustrative.

Ledgerline's marketing team notices that one assistant describes the product from old third-party reviews and never links to Ledgerline's own guides. The site sits behind Cloudflare, and the security team changed bot settings during a busy quarter.

The five-minute check shows Training set to block, which also blocks a mixed-purpose crawler. A custom WAF rule added during a scraping incident challenges several AI user agents. The served robots.txt starts with a Cloudflare managed block that disallows those agents.

Marketing, legal and security meet and agree to allow Search and Agent, keep Training blocked and remove the old WAF rule. They accept that blocking training may reduce search use by the mixed crawler and record that trade-off. A week later, server logs show 200 responses for the crawlers they chose to allow.

How do you tell a block from other problems?

Not being cited has many causes, so confirm the block before changing settings. Look in server logs or Cloudflare's analytics for requests from the crawlers you care about. A 403 or a challenge page means they're being stopped. A 200 means access works and the cause is elsewhere, such as content, authority or rendering. No requests at all can mean the crawler hasn't visited, or that it's blocked before it reaches your origin, in which case only Cloudflare's own logs will show it.

Match user agents against each vendor's published list rather than guessing from names (OpenAI, Anthropic, Perplexity). Vendors often run separate agents for search indexing, user-triggered fetches and training, and a rule that catches one may catch the others. For a step-by-step guide to log analysis, see reading server logs for AI bot visits.

What about robots.txt you already have?

If your origin serves its own robots.txt, Cloudflare's managed content is placed before yours in a single file when Bot Preference Sync is on and a block policy is set (Cloudflare). That means two sets of rules can apply. Read the served file from top to bottom, not the one in your code repository, and check that the combined rules say what you intend. Our view: conflicting directives are hard to reason about; keep your own file short and let one system own AI rules.

Which teams need to be involved?

Security teams usually own Cloudflare and have good reasons to block bots: scraping, load and abuse. Marketing owns visibility. Legal may have views on training and licensing. None of them can decide alone without creating problems for the others. Write a short policy stating which categories you allow, which you block and why, with a named owner, and review it when Cloudflare changes defaults or when you notice a change in AI referral traffic.

How often should you re-check?

Re-check after any Cloudflare announcement about AI bot defaults, after security incidents that led to new rules and at least once a quarter. Cloudflare's defaults have already changed more than once (Cloudflare blog; Cloudflare), and existing domains keep their settings unless someone changes them, so a site can stay on an old configuration for a long time. Add the check to the same calendar as your technical SEO audit.

What do people get wrong when checking?

• Reading only the dashboard. Custom WAF rules and page rules can block bots that the AI bot policy allows.
• Testing with a browser. A browser request isn't a crawler request. Use logs, or test with the crawler's user agent knowing that some vendors verify by IP as well.
• Assuming robots.txt is enforcement. Cloudflare notes robots.txt compliance is voluntary (Cloudflare); it states preferences, while blocks happen in the WAF and AI Crawl Control.
• Blocking training without knowing the side effect. Mixed Search-and-Training crawlers are blocked by any training block (Cloudflare).

What should you record after the check?

Write down the date, the Search, Agent and Training settings, any AI Crawl Control rules per crawler, any custom WAF rules affecting bots and the first lines of the served robots.txt. Add the decision behind each setting and the owner. A one-page record like this makes the next check quick and lets anyone see whether a later drop in AI referrals lines up with a configuration change. Keep it with your other technical audit notes so it isn't lost when people change roles.

Common questions

Does blocking training crawlers hide me from AI search? It can, if an engine uses one mixed-purpose crawler for both. Cloudflare blocks mixed crawlers whenever training is blocked.

Is robots.txt enough to stop crawlers? No. Cloudflare states compliance is voluntary. Use AI Crawl Control or WAF rules to enforce.

Do existing sites get the new defaults? Existing domains keep their settings unless someone changes them, so check rather than assume.

Related

• Reading server logs for AI bot visits
• llms.txt vs robots.txt

Sources

• Cloudflare, Block AI Bots / AI bot policies; robots.txt setting; AI Crawl Control; Verified bots; Content Independence Day blog
• OpenAI, crawlers; Anthropic, crawlers; Perplexity, crawlers

All accessed 30 Oct 2026.