Paywalls, Logins and AI Access

AI search engines can only cite what their crawlers can fetch, so content behind a hard paywall or login is usually invisible to them unless you deliberately expose some of it. For Google, mark paywalled sections with `isAccessibleForFree` structured data so a paywall isn't mistaken for cloaking ([Google](https://developers.google.com/search/docs/appearance/structured-data/paywalled-content)). Our view: keep a strong, citable summary outside the wall.

Facts are from Google, OpenAI and Cloudflare, accessed 30 Oct 2026. Points marked "Our view" are interpretation.

On this page

• What do crawlers actually see?
• How do you mark paywalled content for Google?
• What about user-initiated fetches?
• What is pay per crawl?
• Which approach fits your goal?
• What should a public summary include?
• Does flexible sampling still matter?
• Which pitfalls should you watch for?
• How do you audit your own site?
• Exercise
• Checklist
• From MentionX data: what engines can cite
• Worked example: Ledgerline opens up its benchmark report
• How do you choose what to gate?
• What should a public summary contain?
• How do logins affect documentation?
• How should you treat AI crawlers on gated sections?
• How do you check for leaks?
• How do you measure the trade-off?
• How do you mark paywalled content correctly?
• How do you decide section by section?
• What should you tell stakeholders?
• Common questions
• Related
• Sources

Checked on 31 October 2026. "Our view" marks MentionX opinion. Ledgerline is a fictional company used for illustration.

What do crawlers actually see?

Our view: treat the HTML returned to an anonymous request as what AI crawlers get.

Test with curl -A "OAI-SearchBot" https://example.com/article and look for your key paragraphs. Some crawlers don't run JavaScript; see JavaScript rendering and AI crawlers.

How do you mark paywalled content for Google?

Google's guidance applies only to content you want crawled and indexed (Google):

1. Wrap each paywalled section in a class, e.g. .paywall.
2. Add isAccessibleForFree: false to the article.
3. Add hasPart with a WebPageElement, isAccessibleForFree: false and cssSelector: ".paywall".
4. Use only .class selectors and don't nest content sections.
5. If you don't want content accessible to the browser when served, use a paywall that doesn't send the content.

{
  "@context": "https://schema.org",
  "@type": "NewsArticle",
  "headline": "Article headline",
  "isAccessibleForFree": false,
  "hasPart": {
    "@type": "WebPageElement",
    "isAccessibleForFree": false,
    "cssSelector": ".paywall"
  }
}

Google says this helps it tell paywalled content apart from cloaking, which violates spam policies (Google). Since AI Overviews and AI Mode use Google Search's systems, the same crawling and indexing rules apply (Google).

What about user-initiated fetches?

OpenAI says ChatGPT-User may visit a page when a user asks ChatGPT a question, isn't used for automatic crawling, and because these actions are user-initiated, robots.txt rules may not apply. OAI-SearchBot is the agent to manage for search (OpenAI).

Our view: a user-initiated fetch still hits your paywall like any anonymous visitor. If the content isn't in the HTML, the assistant can't read it. If it is in the HTML but hidden by JavaScript, it may be readable.

What is pay per crawl?

Cloudflare's pay per crawl, in closed beta as of 30 Oct 2026, lets site owners set a price per zone. AI crawlers either present payment intent in request headers for a 200 response or receive HTTP 402 Payment Required with pricing; Cloudflare acts as merchant of record (Cloudflare). Blocking a crawler with WAF or Bot Management overrides the charge.

Our view: it's an option for publishers whose content is the product. For brands seeking visibility, charging crawlers is usually counterproductive.

Which approach fits your goal?

Our view:

What should a public summary include?

Our view: the answer an AI would quote. Give the headline finding with numbers, a date, the method in one or two sentences, who it's for, and a clear title. That gives crawlers something accurate to cite and readers a reason to subscribe.

Does flexible sampling still matter?

Our view: metering choices, such as how many free articles a visitor gets, affect human readers far more than AI crawlers, which usually arrive without cookies. What matters for AI visibility is what the first, anonymous response contains.

Which pitfalls should you watch for?

• Gating pricing or product specs, so AI answers rely on third-party guesses.
• Gated reports with no public findings page.
• Serving full text to Googlebot but a teaser to users, without markup (cloaking risk).
• Blocking all AI bots at the CDN while wanting AI visibility; see Cloudflare AI bot blocking.
• Help centers behind login, which hides answers to common questions.

How do you audit your own site?

Our view: list your 20 most valuable pages for AI answers, such as pricing, comparisons, research and docs. For each, fetch it anonymously with curl, record whether the key facts are present, and note any paywall or login. Then decide per page using the table above. Re-run after site redesigns, because new paywall or consent scripts often change what anonymous visitors receive.

Exercise

Choose five gated assets. For each, write a 100-word public summary with the key finding and a date, publish it on an open page, and check in a month whether AI assistants cite it.

Checklist

• Anonymous fetch checked for key pages
• Paywall markup on indexable paywalled content
• Public summaries for gated assets
• Pricing and help docs public
• CDN bot rules match goals
• Pay per crawl considered only if content is the product

From MentionX data: what engines can cite

Method. Read-only aggregate queries over MentionX scans from 22 August to 29 October 2026 (UTC): 2,217 engine runs (2,154 completed) across 18 anonymised projects and six engines.

Engines cite pages they can reach. In our sample, cited pages were mostly public types such as articles, homepages and docs:

Some engines rarely cite anything, which means public pages need to be clear enough to be described even without a link:

Our view: gated content can still earn visibility, but only through what's public: summaries, abstracts, landing pages and docs. If none of those explain what's behind the gate, engines have nothing to work with.

Limits. A convenience sample of mostly B2B software projects, run through engine APIs, which can behave differently from the consumer apps. Perplexity, Copilot and DeepSeek figures come from a single day (26 October 2026), so treat them as a snapshot, not a trend.

Worked example: Ledgerline opens up its benchmark report

Ledgerline is a fictional accounts-payable tool. Figures are illustrative.

Ledgerline publishes an annual accounts-payable benchmark behind a sign-up form. Engines never cite it, and when buyers ask about AP benchmarks, rivals' public blog posts are quoted instead.

The team adds a public summary page with the method, sample size, dates and the five main findings as a table, and keeps the full report gated. The summary uses clear headings and links to the form. It also checks that the summary isn't blocked in robots.txt and that the report PDF is not linked from public pages without a gate. After two measurement rounds, the summary page is cited for some benchmark prompts in two engines, and form sign-ups hold steady.

How do you choose what to gate?

Gate content that has clear value as a lead or revenue source and that buyers expect to pay or register for: in-depth reports, templates, courses and tools. Keep content public when its main job is to be found and quoted: definitions, how-to guides, product facts, pricing and comparison pages. If a page's purpose is AI visibility, gating it defeats the purpose.

What should a public summary contain?

A clear title, a short description of who the content is for, the method or scope, the main findings or outcomes in plain text, the date and the author. Add a table if the content has key figures. Link to the full version and explain what it adds. A summary like this gives engines something accurate to cite and gives readers a reason to sign up.

How do logins affect documentation?

Some companies put all documentation behind a customer login. That protects detail but hides basic facts, such as supported integrations or system requirements, that buyers ask engines about. A common compromise is a public overview for each feature with detailed steps behind the login. Our view: public docs are among the most useful pages for AI answers about software, so keep at least the overview pages open.

How should you treat AI crawlers on gated sections?

Your robots.txt and access rules decide what crawlers see. If a section is gated, crawlers without credentials can't read it anyway, so blocking it in robots.txt is mostly about tidiness. The bigger decision is about public sections: whether to allow AI crawlers that train models, those that fetch pages for live answers, or both. Make the choice deliberately and record it.

How do you check for leaks?

Gated files sometimes end up public through direct links, sitemaps or cached copies. Search for your report titles, check that PDFs aren't listed in sitemaps and confirm that file URLs require authentication. A leaked PDF that gets cited can undermine both your gate and your control over the content.

How do you measure the trade-off?

Track two things for each gated asset: AI and search visibility of its public pages, and the leads or revenue it generates. Review both after a quarter. If a summary earns citations without hurting sign-ups, extend the approach to other assets. If sign-ups drop sharply, shorten the summary but keep the key facts public.

How do you mark paywalled content correctly?

Google's guidance asks publishers to wrap paywalled sections in a container with a class name and to add structured data that sets isAccessibleForFree to false and points the hasPart property to that class. This tells Google the hidden part is behind a paywall rather than an attempt to show crawlers something different from users. Check the markup with the Rich Results Test and URL Inspection after each template change.

How do you decide section by section?

Make a simple table of your site's sections. For each, note the goal (visibility, leads or revenue), the current access rule and whether engines cite it. Sections whose goal is visibility should be public. Sections whose goal is revenue can stay gated, with a public summary. Review the table each quarter, because the right balance changes as products and buyer habits change.

What should you tell stakeholders?

Explain that gating trades reach for leads. Show which gated assets have public summaries, which are cited in AI answers and how many leads each generates. Then agree the rule for new content: public by default when the goal is visibility, gated with a summary when the goal is leads. A shared rule saves debate on every new asset.

Common questions

Is showing Google more than users cloaking? It can be. Use paywall structured data and follow Google's flexible sampling guidance.

Should I block AI crawlers from gated content? They can't access content behind a login anyway; focus on public sections.

Will a summary page cut sign-ups? Often not, if it explains clearly what the full version adds.

Related

• Markdown versions of pages for LLMs
• Technical GEO audit checklist

Sources

• Google Search Central, paywalled content structured data; AI features
• OpenAI, crawlers
• Cloudflare, What is pay per crawl

All accessed 30 Oct 2026.