JavaScript Rendering and AI Crawlers

Many AI crawlers don't run JavaScript. A December 2024 Vercel and MERJ analysis found none of the major AI crawlers it studied rendered it, including OpenAI's, Anthropic's and Perplexity's bots ([Vercel](https://vercel.com/blog/the-rise-of-the-ai-crawler)). If your main content only appears after client-side JavaScript runs, those crawlers may see an empty page. Server-render or pre-render anything you want cited.

Facts are from Vercel and Google Search Central, accessed 30 Oct 2026. Points marked "Our view" are interpretation. Crawler behaviour can change; re-test periodically.

On this page

• What did the crawler study find?
• What does Google say?
• How do you test what a crawler sees?
• How do you fix it?
• How current is this evidence?
• Which content matters most?
• What goes wrong? (Troubleshooting)
• Exercise
• Checklist
• From MentionX data: which pages are most exposed
• Worked example: Ledgerline's pricing page goes missing
• Why don't most AI crawlers render JavaScript?
• How do you audit a whole site?
• What does the fix cost?
• How does rendering interact with speed?
• What should you tell developers?
• What about content behind interactions?
• How do you keep pages crawler-readable over time?
• What should you check first on a new site?
• Common questions
• Related
• Sources

Checked on 31 October 2026. "Our view" marks MentionX opinion. Ledgerline is a fictional company used for illustration.

What did the crawler study find?

Vercel and MERJ analysed AI crawler traffic across Vercel's network, published 17 Dec 2024 (Vercel):

The study found ChatGPT and Claude crawlers do fetch JavaScript files (11.50% and 23.84% of their requests) but don't execute them. It notes content included in the initial HTML response, such as JSON data or delayed React Server Components, may still be indexed. Microsoft Copilot was excluded because it lacks a unique user agent.

What does Google say?

Google processes JavaScript web apps in three phases: crawling, rendering and indexing, and queues pages for rendering (Google). Google adds that server-side or pre-rendering is still a great idea because it makes sites faster for users and crawlers, "and not all bots can run JavaScript". It also notes rendering might be skipped for pages with a non-200 status code.

How do you test what a crawler sees?

1. Fetch the raw HTML without running scripts:

bash curl -s -A "Mozilla/5.0 (compatible; GPTBot/1.0)" https://www.example.com/page | grep -c "your key phrase" `` A count of 0 means the phrase isn't in the initial HTML.

1. Disable JavaScript in your browser and load the page.
2. Check structured data is in the HTML source, not injected later.
3. Compare with Google's URL Inspection tool rendered view, which reflects Googlebot's rendering.
4. Check logs for which bots fetch the page; see reading server logs for AI bot visits.

Expected result: headings, body copy, prices, author and JSON-LD appear in the curl output.

Spoofing a user agent with curl tests what your server returns for that string; it isn't the real crawler. Some sites vary content by user agent or block unknown bots, so also test with a normal browser user agent.

How do you fix it?

Vercel's recommendations (Vercel):

• Prioritise server-side rendering for critical content: main content, meta information and navigation.
• Use SSR, ISR or SSG to keep content accessible to all crawlers.
• Keep client-side rendering for non-essential features such as view counters, live chat and social feeds.
• Maintain redirects and sitemaps; the study saw high 404 rates from AI crawlers.

Our view: if you pre-render only for bots, make sure the content is identical to what users see; serving different content to crawlers risks being treated as cloaking by search engines.

How current is this evidence?

The Vercel and MERJ figures come from traffic in the month before December 2024. Crawlers evolve, and vendors don't always publish whether their bots render pages. Our view: treat "AI crawlers don't run JavaScript" as the safe default rather than a permanent fact, and design so it doesn't matter: if the content is in the initial HTML, every crawler sees it regardless of rendering. Re-run the curl test after major framework upgrades or CDN changes, since a configuration change can move content from server to client without anyone noticing.

Which content matters most?

Our view: prioritise the things AI answers quote: the opening answer paragraph, headings, tables, prices and plan names, product specs, author and dates, and JSON-LD. Tabs and accordions that load content via JavaScript on click are a common blind spot; render their content in the HTML and toggle visibility with CSS.

What goes wrong? (Troubleshooting)

Exercise

Pick your five most important pages. Run the curl test for a key phrase, the price (if any) and the JSON-LD on each. Record which are missing from the raw HTML and the rendering fix for each.

Checklist

• Main content in initial HTML
• Titles, descriptions and JSON-LD server-side
• Prices and specs not API-only
• Tab and accordion content in HTML
• Redirects and sitemap current
• Re-test after framework upgrades

From MentionX data: which pages are most exposed

Method. Read-only aggregate queries over MentionX scans from 22 August to 29 October 2026 (UTC): 2,217 engine runs (2,154 completed) across 18 anonymised projects and six engines.

Rendering problems hurt most on the pages engines cite. In our sample, the cited pages we could classify broke down like this:

Documentation, pricing and product pages are often built as client-side apps or load key details from an API, which is exactly where raw HTML can be empty. Engines also differ in how often they cite at all:

Our view: if an engine that cites often never links to your docs or pricing page, check whether those pages render content in the initial HTML before looking for other causes.

Limits. A convenience sample of mostly B2B software projects, run through engine APIs, which can behave differently from the consumer apps. Perplexity, Copilot and DeepSeek figures come from a single day (26 October 2026), so treat them as a snapshot, not a trend.

Worked example: Ledgerline's pricing page goes missing

Ledgerline is a fictional accounts-payable tool. Figures are illustrative.

Ledgerline notices that assistants describe its plans vaguely or with old prices. The pricing page looks fine in a browser, but it's a client-side component that fetches plans from an API after load.

A raw HTML fetch with a crawler user agent returns the page shell, navigation and footer, but no plan names, no prices and no FAQ. The JSON-LD is injected by a tag manager, so it's missing too.

The team switches the pricing page to static generation with a rebuild whenever prices change, renders the FAQ server-side and moves the JSON-LD into the server template. The curl test now finds every plan name and price. The team adds the test to its release checklist so a future redesign can't reintroduce the problem.

Why don't most AI crawlers render JavaScript?

Rendering costs far more than fetching: the crawler has to run a browser, load scripts and wait for content to appear. Google built that infrastructure over years for search, and processes JavaScript pages in separate crawling, rendering and indexing phases (Google). The Vercel and MERJ study found that most AI crawlers it observed fetched JavaScript files but didn't execute them (Vercel). Our view: vendors may change this, but building for crawlers that read only HTML is the safe default.

How do you audit a whole site?

Start with templates rather than pages. Most sites have a few: homepage, product, pricing, docs, blog post, comparison and glossary. Run the raw HTML test on one example of each, checking for the main heading, the opening paragraph, a table, a price if relevant and the JSON-LD. Any template that fails will fail on every page built from it, so one fix solves many pages.

Then check the parts within templates that often load late: tabs, accordions, "read more" sections, reviews widgets, pricing toggles and related-content blocks. Content inside them may be absent from the HTML even when the rest of the page renders.

What does the fix cost?

It depends on your stack. Modern frameworks such as Next.js, Nuxt, SvelteKit and Astro support server-side rendering and static generation, and switching a page may be a configuration change. Older single-page apps may need a pre-rendering service as a bridge while you migrate. Our view: fix the pages engines cite most first: guides, docs, pricing and comparisons. Interactive tools and dashboards behind login can stay client-side.

How does rendering interact with speed?

Server-rendered and static pages usually deliver content faster as well, because the browser doesn't have to wait for scripts to fetch data. Crawlers with tight time limits benefit from that. See page speed and AI retrieval for more on why fast responses matter.

What should you tell developers?

Give them a short, testable requirement: the main content, headings, prices, author, dates and JSON-LD for these templates must be present in the initial HTML response. Provide the curl test and the list of key phrases per template. Our view: framing it as a test that passes or fails avoids long debates about frameworks and keeps the requirement clear when the stack changes.

What about content behind interactions?

Content that only appears after a click, such as a tab or a "show more" button, is often loaded on demand. If it matters for answers, include it in the HTML and use CSS to hide or show it. Visitors get the same experience, and crawlers that read only HTML can still see it.

How do you keep pages crawler-readable over time?

Rendering problems tend to come back. A redesign moves content into a new component, a marketing tool injects a widget, a framework upgrade changes defaults or a team adds a client-side pricing toggle. Each can quietly remove content from the initial HTML.

Protect against this with three habits. First, keep a list of key phrases per template and test them automatically on every deploy. Second, re-run the full audit after any framework upgrade or redesign. Third, watch your server logs for AI crawler requests to important pages and for spikes in 404s, which the Vercel and MERJ study saw at high rates from AI crawlers (Vercel). Our view: a failing test in a build pipeline is far cheaper than discovering months later that assistants have been describing an empty page.

What should you check first on a new site?

Start with the homepage, the main pricing or product page and your three most-visited guides. Fetch each without JavaScript and confirm the main heading, the opening answer, key facts and JSON-LD are present. If all five pass, rendering is unlikely to be your main problem, and you can move on to crawler access and content quality.

Common questions

Does Google render JavaScript? Yes. Google processes JavaScript apps in crawling, rendering and indexing phases, though it still recommends server-side or pre-rendering.

Is serving pre-rendered pages only to bots allowed? Keep the content identical to what users see; different content risks being treated as cloaking.

Do AI crawlers read JSON-LD injected by a tag manager? Crawlers that don't run JavaScript won't see it. Output JSON-LD server-side.

Related

• Page speed and AI retrieval
• Markdown versions of pages for LLMs

Sources

• Vercel and MERJ, The rise of the AI crawler, 17 Dec 2024
• Google, JavaScript SEO basics; URL Inspection tool

All accessed 30 Oct 2026.