Page Speed and AI Retrieval

Page speed matters to AI retrieval mainly through crawling: Google says that when a site's response times increase or it returns 5xx or 429 errors, its crawl capacity limit goes down and it crawls less ([Google](https://developers.google.com/crawling/docs/crawl-budget)). AI vendors don't publish speed thresholds or fetch timeouts. So fix server response time and errors first, then Core Web Vitals for users.

Facts are from Google Search Central and Vercel, accessed 30 Oct 2026. Points marked "Our view" are interpretation.

On this page

• What does Google document about speed and crawling?
• What are the Core Web Vitals targets?
• What do AI vendors say?
• What should you fix first?
• How do you measure bot-facing speed?
• What goes wrong? (Troubleshooting)
• How do you keep it fast over time?
• Is there evidence that speed changes AI citations?
• Exercise
• Checklist
• From MentionX data: why access matters more than speed scores
• Worked example: Ledgerline speeds up its docs for bots
• Which speed problems hurt bots most?
• How is bot speed different from user speed?
• How do you test what a crawler receives?
• Should you treat AI crawlers differently?
• How does a CDN help?
• How do you prioritise speed work?
• How do caching and freshness interact?
• What about images and media?
• How do you keep speed from regressing?
• When is speed not the problem?
• Common questions
• Related
• Sources

Checked on 31 October 2026. "Our view" marks MentionX opinion. Ledgerline is a fictional company used for illustration.

What does Google document about speed and crawling?

Google's crawl budget guide says crawlers calculate a crawl capacity limit to avoid overloading servers (Google):

• If the site responds consistently and response times, including latency and time to first byte, stay stable or improve, the limit goes up.
• If the site slows down or returns server errors (5xx) or rate-limiting signals (429), the limit goes down and Google crawls less.

Google's Gemini uses Googlebot's infrastructure, according to Vercel and MERJ's crawler analysis (Vercel), so slow responses affecting Googlebot plausibly also affect what Google's AI features can draw on. Our view: that's an inference, not something Google states for AI Overviews specifically.

What are the Core Web Vitals targets?

Source: Google. Google says Core Web Vitals are used by its ranking systems, there's no single page experience signal, and it always seeks to show the most relevant content even if page experience is sub-par (Google).

What do AI vendors say?

OpenAI, Anthropic and Perplexity document their crawler user agents and robots.txt controls, but we found no published speed thresholds or fetch timeouts for their bots (OpenAI, Anthropic, Perplexity). Our view: don't trust claims of a specific "AI timeout" unless they cite a vendor document. What you can measure is how your server responds to these bots in your own logs.

What should you fix first?

Our view, in priority order:

1. Server errors and rate limits. 5xx and 429 responses directly reduce Google's crawling (Google). Check that bot traffic isn't being blocked by over-aggressive rate limits you didn't intend.
2. Time to first byte. Cache HTML at the edge, use static generation where possible, and avoid slow database calls on page render.
3. Content in the initial HTML. Bots that don't run JavaScript only see the first response; see JavaScript rendering and AI crawlers.
4. Page weight. Large HTML with inlined scripts slows every fetch.
5. Core Web Vitals for real users, which Google uses in ranking.

How do you measure bot-facing speed?

1. Logs: record response time and status code per request, then filter by AI user agents. See reading server logs for AI bot visits.
2. Search Console Crawl Stats: average response time and host status for Googlebot.
3. curl timing:

bash curl -o /dev/null -s -w "TTFB %{time_starttransfer}s total %{time_total}s status %{http_code}\n" https://www.example.com/page

1. Core Web Vitals report in Search Console for field data (Google).

What goes wrong? (Troubleshooting)

How do you keep it fast over time?

Our view: add a performance budget to your release process, such as a maximum HTML size and TTFB for key templates, and alert on rising 5xx or 429 rates to bot user agents. Review the CDN and WAF rules after any security change, because a new bot-protection setting is a common cause of sudden crawl drops. Track Crawl Stats monthly alongside AI visibility so you can spot whether a slowdown preceded a change in citations.

Is there evidence that speed changes AI citations?

We haven't found published, controlled studies showing faster pages get cited more by AI assistants. Our view: speed is a hygiene factor: it won't make weak content citable, but slow responses and errors can stop strong content being fetched at all.

Exercise

Pull one week of logs. For each AI user agent, compute the share of 5xx and 429 responses and the median response time. Compare with human traffic and list any fixes.

Checklist

• No sustained 5xx or 429 to wanted bots
• HTML cached at the edge where possible
• Main content in initial HTML
• Crawl Stats response time monitored
• Core Web Vitals in "Good"

From MentionX data: why access matters more than speed scores

Method. Read-only aggregate queries over MentionX scans from 22 August to 29 October 2026 (UTC): 2,217 engine runs (2,154 completed) across 18 anonymised projects and six engines.

We don't measure page speed in our answer data, so we can't say speed changes citation rates. What our data does show is which kinds of pages get cited, and those pages must be fetched to be used:

Our view: the pages most cited in our sample, such as product, documentation and comparison pages, are often the ones heavy scripts slow down. Make sure those load quickly and fully without JavaScript before you chase lab scores on the blog.

Limits. A convenience sample of mostly B2B software projects, run through engine APIs, which can behave differently from the consumer apps. Perplexity, Copilot and DeepSeek figures come from a single day (26 October 2026), so treat them as a snapshot, not a trend.

Worked example: Ledgerline speeds up its docs for bots

Ledgerline is a fictional accounts-payable tool. Figures are illustrative.

Ledgerline's server logs show AI crawlers often time out on its documentation pages, which are rendered in the browser and load a large script bundle. The team switches the docs to server-side rendering, adds caching at the edge and trims third-party scripts.

Average server response time drops, and log timeouts for crawler requests fall to near zero. The team doesn't claim this raised citations. It does confirm that crawlers now receive the full page text, and it keeps a weekly check on response times for bot requests.

Which speed problems hurt bots most?

• Slow server response. If the first byte takes seconds, crawlers may give up or fetch fewer pages.
• Content that needs JavaScript. Many AI crawlers don't run scripts, so slow or failed rendering means missing text.
• Large pages. Very heavy HTML or inline data can be cut off.
• Errors under load. 5xx errors during traffic spikes tell crawlers your site is unreliable.
• Redirect chains. Each hop adds delay and a chance to fail.

How is bot speed different from user speed?

Core Web Vitals measure what people experience in a browser: loading, responsiveness and layout shifts. Bots care mainly about how fast the server responds and whether the HTML contains the content. A page can score well for users, because it shows a fast skeleton, and still serve bots an empty shell. Check both views.

How do you test what a crawler receives?

Fetch the page with a simple HTTP client, without running JavaScript, and time the response. Check that the main heading, the opening answer and key facts are in the HTML. Repeat for your most important templates: product, pricing, documentation, comparison and blog. If any template is slow or empty, fix that template, which fixes every page built on it.

Should you treat AI crawlers differently?

Serve them the same content as users. Don't create special fast versions for bots that differ from what people see; that risks cloaking. You can, however, make sure caching and rate limits don't block legitimate crawlers. If you rate-limit bots, check that the limits aren't so tight that crawlers get errors on your key pages.

How does a CDN help?

A content delivery network caches pages close to the requester, which cuts response time for both people and bots. It can also absorb traffic spikes that would otherwise cause errors. Check your CDN's bot settings: some block or challenge unknown bots by default, which can stop AI crawlers you want to allow.

How do you prioritise speed work?

Start with pages that matter most for AI answers: pricing, product, comparison and documentation. Fix server response time and rendering on those templates. Then work through the rest of the site. Our view: a modest, reliable response on every key page is worth more than a perfect score on a few.

How do caching and freshness interact?

Long cache times make pages fast but can serve stale facts, such as old prices, to crawlers. Set shorter cache lifetimes on pages whose facts change often, and purge the cache when you update pricing or plans. Keep longer lifetimes for stable guides. Check that your sitemap's lastmod dates match real content changes, so crawlers know when to refetch.

What about images and media?

Large images slow pages for people but matter less for text crawlers, which usually fetch the HTML and skip media. Still, compress images and lazy-load those below the fold, because user speed affects search rankings and AI features in Google Search draw on pages that rank. Give important images descriptive alt text so the meaning survives when the image isn't loaded.

How do you keep speed from regressing?

Add a check to your release process. Before each deploy, fetch your key templates without JavaScript and time them. Fail the check if response time or page size crosses a threshold you set from your own baseline. Track third-party scripts in a list with an owner for each; marketing tags are a common cause of slow pages, and nobody removes them unless someone owns the list. Our view: review the tag list every quarter and remove anything no one can justify.

When is speed not the problem?

If crawlers fetch your pages quickly and the HTML holds the content, but you still aren't cited, look elsewhere: content quality, coverage of buyer questions and third-party mentions. Fix those first; speed work will add little until they improve.

Common questions

Does page speed affect AI citations? There's no public evidence of a direct effect. Speed affects whether crawlers can fetch your content reliably.

Do AI crawlers run JavaScript? Many don't. Put key content in the server-rendered HTML.

Which metric should I watch for bots? Server response time and error rates for crawler requests in your logs.

Related

• JavaScript rendering and AI crawlers
• Technical GEO audit checklist

Sources

• Google, crawl budget management; Core Web Vitals; page experience
• Vercel and MERJ, The rise of the AI crawler
• OpenAI, crawlers; Anthropic, crawler docs; Perplexity, crawlers

All accessed 30 Oct 2026.