Why AI Answers Differ Every Time (and How to Measure Anyway)

AI answers differ between runs because language models sample their wording, search results change, and products may personalise by location, language or memory. You can still measure visibility reliably by running a fixed prompt set several times per engine, counting how often your brand is mentioned, cited or recommended, and reporting those rates as ranges.

Why do AI answers change?

Google notes that AI Overviews and AI Mode may use different models and techniques, so the responses and links they show will vary (Google).

Why is one screenshot misleading?

A single answer is one sample. If your brand appears in 40% of answers for a prompt, a single test will show you missing more often than present. Teams that judge visibility from one screenshot overreact to noise: celebrating one good run, or panicking at one bad one.

How do you turn answers into rates?

Instead of asking "Are we in the answer?", ask "How often are we in the answer?"

1. Fix a prompt set: 25–100 real buyer questions.
2. Run each prompt several times on each engine.
3. Record, for each answer: mentioned? cited? recommended? position? accurate?
4. Calculate rates: mentions ÷ answers.
5. Compare rates over time and against competitors.

How do you know if a change is real?

Use a confidence interval. For a proportion, a standard approach gives a range around the observed rate that narrows as the sample grows (NIST).

Illustrative. If you're mentioned in 20 of 100 answers (20%), a 95% interval is roughly 13% to 29%. If the next month shows 24 of 100, the ranges overlap heavily, so that isn't clear evidence of improvement. A move to 40 of 100 (about 31% to 50%) is.

Rules of thumb:

• Bigger samples give narrower ranges.
• Overlapping ranges mean you can't be confident a change is real.
• Trends over several months are more reliable than one comparison.

How should you control context?

Keep conditions consistent between runs:

• Same engines and product versions where possible
• Logged-out or clean sessions to limit memory effects
• Same location and language settings
• Same prompt wording
• Record the date and any known model updates

See Personalization and memory and Location and language effects.

What does a repeatable routine look like?

How do you report variation to stakeholders?

• Report rates, not individual answers
• Show ranges alongside each rate
• Use a labelled example answer to illustrate, not prove
• Note model updates that might explain sudden changes

How many runs are enough?

There's no universal number, but you can work out the precision you'll get. The width of a 95% range for a proportion shrinks roughly with the square root of the sample size (NIST).

Illustrative ranges for an observed rate of 20%:

To halve the width of the range you need about four times as many answers. For most teams, 50–100 answers per engine for each prompt cluster each month gives ranges narrow enough to spot real changes. Spread those runs across many prompts rather than repeating a few, so the results reflect your whole market.

What does the research say about variation?

The GEO paper sampled five responses per query at a temperature of 0.7 and averaged results over five random seeds, explicitly "to reduce statistical deviations" (arXiv). Even under controlled lab conditions, researchers treated single answers as unreliable. Commercial products add search changes, personalisation and model updates on top of that.

Should you use the API or the consumer app?

APIs make repeat runs cheap and controllable, but they may not match what customers see: consumer apps can add web search, memory and different system settings. Use the consumer products, or tools that collect from them, for visibility reporting. Use APIs for quick internal tests, and label which source each result comes from.

A quick example

Illustrative. A marketing lead sees a competitor recommended in a ChatGPT answer and asks for an urgent response. The team checks the monthly data: across 120 answers, the company's recommendation rate is 26% (about 19–35%) versus the competitor's 29% (about 21–38%). The ranges overlap, so the brands are roughly level. The team keeps its plan rather than reacting to one screenshot.

What mistakes should you avoid?

• Judging visibility from one run
• Changing the prompt set every month
• Mixing logged-in and logged-out results
• Reporting precise percentages from tiny samples
• Ignoring model updates when explaining changes

Checklist

• [ ] Fixed prompt set
• [ ] Several runs per prompt per engine
• [ ] Rates calculated, not single answers
• [ ] Ranges reported
• [ ] Conditions kept consistent
• [ ] Model updates logged

Related

• How AI search engines build an answer
• How often should you check AI visibility?

Sources

• Google Search Central, AI features and your website
• NIST/SEMATECH, Confidence intervals for proportions
• Aggarwal et al., GEO: Generative Engine Optimization, KDD 2024