Screaming Frog for Technical GEO

#01What does Screaming Frog cost?

Screaming Frog's pricing page lists JavaScript rendering, custom extraction and "Crawl with OpenAI & Gemini" among the paid SEO Spider features (Screaming Frog). Check the live page for the current licence price and currency.

#02How do you check robots.txt rules for AI crawlers?

By default the SEO Spider obeys robots.txt and uses its own user-agent string. Under Configuration > User-Agent you can switch the user-agent of its requests, which also changes which robots.txt directives it follows (Screaming Frog). The custom robots.txt feature lets you test directives and see which URLs are blocked or allowed for the selected user-agent (Screaming Frog).

Fictional workflow:

1. Set the user-agent to the string a crawler publishes, for example OAI-SearchBot from OpenAI's crawler page or PerplexityBot from Perplexity's crawler page.
2. Crawl the site with "Respect robots.txt" on.
3. Look at URLs blocked by robots.txt. If key product, pricing or help pages are blocked, that bot can't fetch them.
4. Repeat for each AI user-agent you care about.

This tests your rules, not the bot. A matching user-agent string doesn't make your crawl behave like OpenAI's or Perplexity's systems, and some sites' firewalls treat verified bots differently from spoofed ones. See configuring robots.txt for AI crawlers and the list of AI crawler user-agents.

#03How do you find content that only appears after JavaScript runs?

The SEO Spider has rendering modes. In "Text Only" mode it crawls and extracts from the raw HTML only, ignoring client-side JavaScript; JavaScript rendering is set under Configuration > Spider > Rendering (Screaming Frog).

Fictional workflow: crawl once in Text Only mode and once with JavaScript rendering, then compare word count, headings and links per URL. Pages where the main content, prices or FAQs only appear in the rendered crawl rely on JavaScript.

Why that matters: Google documents how it renders JavaScript (Google), but not every AI crawler documents doing the same. Our view: if the important facts on a page are only in rendered HTML, treat that as a risk for AI visibility and move them into the server-rendered HTML. See JavaScript rendering and AI crawlers.

#04How can custom extraction help GEO audits?

Custom extraction lets you scrape data from pages using XPath, CSS paths or regex during a crawl (Screaming Frog). For GEO, useful extractions include:

Custom search is a related feature: it finds pages that do or don't contain a given text or regex in the HTML, page text or an element (Screaming Frog). Use it to find pages missing a disclaimer, a price or an outdated product name. See schema markup for AI search.

#05What do the AI features in the SEO Spider do?

Screaming Frog lists "Crawl with OpenAI & Gemini" as a paid feature (Screaming Frog). The user guide also describes an embeddings feature under Configuration > Content > Embeddings. It uses vector embeddings from an AI provider (OpenAI, Gemini or Ollama, with your own API key) to find semantically similar pages and low-relevance content, power semantic search, and draw a content cluster diagram (Screaming Frog).

Our view: embeddings help spot near-duplicate pages that compete for the same questions and topic gaps in a cluster. Treat them as a way to prioritise, not as a measure of AI visibility. API calls are billed by the provider you choose.

#06How do you see which AI bots actually visit?

The SEO Spider shows what bots could fetch; server logs show what they did fetch. Screaming Frog's Log File Analyser lets you upload log files, verify search engine and AI bots, see which URLs were crawled and analyse bot behaviour (Screaming Frog).

Fictional questions to answer from logs:

• Which AI user-agents visited in the last 30 days, and how often?
• Did they fetch your key pages, or mostly assets and old URLs?
• Which requests returned errors or redirects?
• Are requests claiming to be an AI bot coming from the operator's published IP ranges? Perplexity, for example, publishes its bot IPs (Perplexity).

See reading server logs for AI bot visits and log analyzers for AI bots.

#07What can't Screaming Frog tell you?

It doesn't tell you whether ChatGPT, Perplexity or Google's AI features mention or cite you. That needs prompt tracking. See best AI visibility tools or build your own tracker.

#08From MentionX data: which pages engines citeOriginal data

Method. Read-only aggregate queries over MentionX scans from 22 August to 29 October 2026 (UTC): 2,217 engine runs (2,154 completed) across 18 anonymised projects and six engines.

A technical audit is about making pages readable. Citations then come from many page types:

And engines differ in how often they show citations at all:

Our view: a crawl tells you whether pages like these are accessible and well structured. It can't tell you whether they're cited, so pair it with prompt tracking.

Limits. A convenience sample of mostly B2B software projects, run through engine APIs, which can behave differently from the consumer apps. Perplexity, Copilot and DeepSeek figures come from a single day (26 October 2026), so treat them as a snapshot, not a trend.

#09Worked example: Ledgerline audits its docs for AI crawlers

Ledgerline is a fictional accounts-payable tool. Figures are illustrative.

Ledgerline's SEO lead crawls the marketing site and docs with a desktop crawler. She checks three things.

First, robots.txt. She tests the rules against the user agents of the AI crawlers the company wants to allow and finds the docs folder blocked by an old rule meant for a staging path. Second, rendering. She compares raw and rendered HTML and finds the pricing table only appears after JavaScript runs. Third, structured data. A custom extraction pulls each page's schema type, showing that half the help articles have none.

She fixes the robots rule, asks engineering to server-render the pricing table and adds basic Article markup to help pages. A month later, the team's server logs show AI crawlers fetching docs pages they hadn't fetched before. All figures and findings here are illustrative.

#10How should you scope an audit for GEO?

Start with the pages you want engines to read: product pages, pricing, docs, comparison pages and your best guides. Crawl those first rather than the whole site. Add the rest once the key templates are clean. Our view: fixing one template often fixes hundreds of pages, so look for patterns, not single URLs.

#11Which issues matter most?

Blocked pages, content that only appears after JavaScript, missing or wrong canonical tags, broken internal links and thin pages without text. Each of these can stop a crawler from reading or trusting a page. Fix access issues first, because nothing else matters if a page can't be fetched.

#12How do you report findings to engineering?

Group issues by template and give each a clear example URL, what's wrong, why it matters and the fix. Engineers prefer one ticket per root cause, not a list of thousands of URLs. Include how you'll check the fix.

#13How often should you crawl?

Monthly works for most sites, plus a crawl after any major release or migration. Compare each crawl with the last to catch regressions, such as a new robots rule or a template that dropped its structured data.

#14What should you check after fixing issues?

Recrawl the affected pages, confirm the fix in raw HTML, and check server logs over the following weeks for crawler visits. Then watch your prompt set for changes in citations, remembering that many other factors affect them.

#15How does a crawler fit alongside other tools?

A crawler shows what's on your site. Server logs show who visits. Search Console shows how Google sees pages. Prompt tracking shows whether engines mention or cite you. Each answers a different question, and none replaces the others.

#16How do you check structured data at scale?

Use custom extraction or the structured data report to list each page's schema types, then compare with what each template should have. Look for pages missing markup, pages with errors and markup that doesn't match visible content. Spot-check a few URLs in Google's Rich Results Test or the Schema Markup Validator to confirm.

#17How do you find thin pages?

Sort crawled pages by word count and look at the lowest. Some will be fine, such as contact pages. Others will be templates with little text, such as tag pages or empty category pages. Decide whether to improve, merge or noindex them. Our view: thin pages rarely get cited, and they dilute crawl attention.

#18What about internal linking?

Check crawl depth and inlinks for your most important pages. A key guide buried five clicks deep with two internal links is hard for any crawler to find. Add links from hubs, navigation and related articles.

#19How do you share crawl results with non-technical colleagues?

Summarise the top three issues in plain language, with one example page each and the expected benefit of fixing them. Keep the full export for the engineers who need it.

#20Common questions

Can Screaming Frog tell us if ChatGPT cites us? No. It audits your site. Use prompt tracking to see citations.

Do we need the paid version? The free version has a crawl limit. Larger sites and features such as custom extraction need a licence; check the current pricing page.

How do we test AI crawler access? Test robots.txt rules against each crawler's user agent, then confirm visits in your server logs.

Should we crawl with JavaScript rendering on? Crawl both ways and compare, so you can see what content depends on JavaScript.