Running a Baseline Scan

A baseline scan is your first full measurement. You run your prompt set from Lesson 5 several times in each engine you care about, score the answers for you and your main competitors, and write the method down. Every later claim of "we improved" or "we dropped" is measured against it, so the baseline has to be repeatable. Lesson 6 of GEO 101 covers which engines, competitors and run counts to choose, the method note, the four core metrics, how big a change has to be before you believe it, and what MentionX data says about mixing branded and unbranded prompts.

In this lesson

• What a baseline is for
• Which engines to include
• Which competitors to track
• How many runs per prompt
• Manual sheet or tool
• The method note
• The four core metrics
• From MentionX data: branded prompts distort the average
• How big a change is real?
• Which cells to trust
• Worked example: Ledgerline Baseline v1
• What can go wrong
• Sharing the baseline
• Exercise
• Quiz
• Sources

Anything marked "Our view" is our opinion. Ledgerline is the fictional company used throughout this course.

What a baseline is for

The manual check in Lesson 4 gave you a sketch. The baseline gives you a reference point. It has three jobs:

1. Measure where you stand on each engine and intent, with enough answers that the numbers mean something.
2. Show where competitors stand, because a 25% mention rate means one thing if the leader has 30% and another if it has 70%.
3. Make the next scan comparable. If the method changes, so do the numbers, and you won't know whether the engines moved or you did.

The third job is the one teams skip. Our view: a slightly imperfect baseline you can repeat exactly is worth more than a perfect one you can't.

Which engines to include

Include the engines your buyers use, and treat each mode as its own engine.

Three or four is plenty for a first baseline. Adding engines multiplies the work. It's better to measure three properly than six badly.

Which competitors to track

Track three to five: the ones you lose deals to, plus one category leader. Ask sales who comes up in late-stage deals. That's usually a different list from the one marketing would pick.

For each, write every name variant you'll match: "Tool A", "ToolA", "toola.com", and old product names. Inconsistent matching is the most common reason competitor numbers look odd.

How many runs per prompt

Answers vary between runs, so one run per prompt measures luck. Three runs per prompt per engine is a practical minimum:

20 prompts × 4 engines × 3 runs = 240 answers.

Run the whole baseline within a few days. Spread across weeks, an engine update can land halfway through and you won't be able to tell.

Manual sheet or tool

Either works for a baseline. The comparison is covered in manual vs automated tracking. Our view: if you use a tool, still read 20 to 30 of the raw answers yourself. A dashboard number is only as good as the extraction behind it.

The method note

Write it before you start, and keep it with the results.

• Prompt set name, version and count, with the intent mix
• Engines and modes (search on/off, AI Overviews vs AI Mode, model if known)
• Market, language and location
• Logged in or out, account type, memory setting
• Runs per prompt, and API sampling settings if relevant
• Competitors and every name variant
• Scoring rules (link to your Lesson 3 rules)
• Scan dates and time zone
• Anything unusual (a known engine outage, a site migration that week)

Illustrative template:

Anyone should be able to repeat the scan from this paragraph alone.

The four core metrics

Calculate each overall, by engine and by intent, and put the number of answers beside every figure. "Mention rate 25% (n=240)" is a result. "25%" on its own isn't. See AI share of voice for the variants of that metric.

Then calculate the same four for each competitor. Also list the 10 most-cited domains and whether each one mentions you.

From MentionX data: branded prompts distort the average

Method. We ran read-only aggregate queries over completed MentionX runs from 22 August to 29 October 2026 (UTC): 2,154 runs across 18 projects, mostly B2B software, engines via API with web search or grounding on. Each prompt carries an intent label. "Mentioned" means our extraction found the tracked brand in the answer. Brands are anonymised.

How to read it. Brand-specific prompts name the brand by design, and comparison prompts usually include it in the question ("X vs Y"). Put them in the overall average and the headline number rises, telling you nothing about discovery. The prompts that show whether buyers find you, best-of, recommendation and problem prompts, sat between 6% and 29%.

Problem prompts were the hardest by far. That fits Lesson 1's point that buyers describing a problem rarely get a vendor list.

Caveats. Intent labels come from each project's prompt set, and the 18 projects aren't a random sample of the market. Use the pattern, not the exact figures.

How big a change is real?

A rate from a sample has a margin of error. For a proportion, a common rough 95% margin is ±1.96 × √(p × (1 − p) ÷ n). The NIST/SEMATECH handbook covers the method for proportions (NIST).

This treats every answer as independent, which repeated runs of the same prompt aren't, so the true uncertainty is a little larger. The practical rules:

• At 60 answers per engine, only differences of 15 points or more are worth acting on.
• A change has to show up in two scans in a row and in more than one prompt before you call it real.
• Report the n beside every rate so nobody over-reads a small cell.

Which cells to trust

Not every cell in your tables deserves the same weight. Before you share the baseline, mark each figure:

• Solid: 100+ answers and a gap of 10 points or more from the comparison. Fine to report as a finding.
• Indicative: 40–100 answers, or a smaller gap. Report it as "appears to", and recheck next scan.
• Too small: under 40 answers. Show the count ("3 of 18"), not a percentage.

Percentages on tiny cells are the most common way baseline reports mislead people. "17%" sounds like a measurement. "3 of 18" looks like the small sample it is.

Worked example: Ledgerline Baseline v1

Illustrative example. Ledgerline, its competitors and every number below are invented.

Ledgerline ran 20 unbranded prompts across three engines, three runs each: 180 answers. Six branded prompts were scored separately.

Branded prompts (n=54): 100% mentioned, but 11 answers quoted an outdated entry price.

What Priya wrote down:

• Tool A dominates recommendations. That 24-point gap in recommendation rate is well outside the margin of error.
• Problem prompts are close to zero. That confirms the manual check.
• Ledgerline vs Tool B on mention rate (28% vs 39%) is an 11-point gap at n=180. It's probably real, but she flagged it to recheck next month.
• The pricing error is a branded-prompt accuracy issue, and it goes into the Lesson 8 plan regardless of rates.

Had she blended the branded prompts into the overall number, the mention rate would have read 45% and looked healthy.

What can go wrong

• Changing engine settings or prompts partway through
• Mixing logged-in and logged-out runs
• Spreading the scan over weeks
• Not saving raw answers, which makes later investigation impossible
• Inconsistent competitor name matching
• Rerunning the baseline because you don't like the result

If you find a genuine method mistake, document it, run a corrected baseline and keep both versions. Don't silently overwrite.

Sharing the baseline

Send a one-page summary:

1. The method note
2. The four metrics, overall and by engine, with n
3. The intent table
4. Each competitor's numbers alongside yours
5. Three early observations, worded as observations rather than conclusions

Keep the raw answers linked for anyone who wants to check. Treat this scan as the fixed reference for at least the next three monthly scans. Lesson 7 turns it into findings.

Exercise

1. Write your method note first, including your definition of "improved".
2. Run the baseline, in a tool or a manual sheet, within one week.
3. Calculate the four metrics overall, by engine and by intent, with n, for you and each competitor.
4. List the 10 most-cited domains. Save everything as "Baseline v1".

Quiz

1. How many answers do 20 prompts × 4 engines × 3 runs give? 240.

2. Why report branded prompts separately? They mention you almost every time (100% in MentionX data), which inflates the overall rate.

3. Why track AI Mode separately from AI Overviews? Answers and links can differ between them.

4. At 60 answers per engine, roughly how precise is a 30% rate? About ±12 points.

5. When should you call a change real? When it's larger than the margin and repeats across two scans and more than one prompt.

Next: Lesson 7: Reading your results.

Sources

• Google Search Central, AI features and your website
• OpenAI Help Center, ChatGPT search
• Google Cloud, Adjust parameter values
• NIST/SEMATECH e-Handbook of Statistical Methods, proportions
• MentionX aggregate scan data, 22 August to 29 October 2026 (UTC), 2,154 engine runs, 18 projects. Anonymised and read-only.