GuidePractitioner10 min readPart of GEO Practitioner

GEO Experiment Design: Testing Changes

How to test whether a GEO change improved AI visibility: hypotheses, test and control pages, baselines, timing and honest analysis for noisy data.

Dipuck Jones
Written by Founder of MentionX · 13+ years in B2B SaaS growth and SEO
ReviewedUpdated
Short answer

A GEO experiment compares AI visibility for pages or topics you changed against similar ones you didn't, before and after the change, so you can separate your effect from background shifts in AI answers. You can't randomly split users of ChatGPT, so the usual design is a controlled before-and-after comparison. Done carefully, it turns opinions about "what works in GEO" into evidence for your own site.

Key takeaways
  • A GEO experiment changes one thing on a group of pages and compares their AI visibility with a similar control group.
  • Write the hypothesis, metric, threshold and timeline before you change anything.
  • Run each prompt several times per period: in MentionX data, answers changed between runs for 11 of 28 prompt-engine pairs.
  • Check the change has been crawled and indexed before judging it, and allow for engines that rarely cite sources.
  • Report results with ranges and sample sizes, and record tests that showed no effect.

Checked on 31 October 2026. "Our view" marks MentionX opinion. Ledgerline is a fictional company used for illustration.

01Why is GEO testing harder than CRO testing?

FactorWebsite A/B testGEO test
Who you randomiseVisitorsCan't randomise AI users
OutcomeYour own conversion dataSampled AI answers
NoiseModerateHigh: answers vary between runs
External changesRare during testModel updates can shift everything
DelayImmediateWait for recrawl and reprocessing

Those differences don't make testing impossible. They mean you need controls, enough observations and patience.

02Step 1: How do you write a testable hypothesis?

Use the format: If we [change], then [metric] for [prompt group] will [direction] by at least [amount], because [reason].

Illustrative: "If we add a comparison table and a dated pricing summary to our five integration pages, then the recommendation rate for integration prompts in ChatGPT will increase by at least 10 points within eight weeks, because answers currently cite outdated third-party pages for pricing."

A good hypothesis names one change type. If you rewrite copy, add schema and build links at once, you won't know what worked.

03Step 2: How do you choose test and control groups?

Pick pages or topics that are similar in traffic, age, type and current visibility. Change only the test group.

GroupPagesPrompts
Test5 integration pages (changed)40 prompts about those integrations
Control5 similar integration pages (unchanged)40 prompts about the control integrations

If both groups rise, an engine-wide change is the likely cause. If only the test group rises, your change is the more plausible explanation. This is a simple form of difference-in-differences: compare the change in test with the change in control.

04Step 3: How do you set the baseline?

  • Run the full prompt set for both groups for at least two to four weeks before the change.
  • Record engine, model where visible, location and date for every run.
  • Check the baseline is stable; if it swings wildly, you need more observations before testing.

05Step 4: How do you know the change has been seen?

Publishing isn't the same as being processed. Before you start counting:

  1. Check server logs for visits from relevant crawlers to the changed URLs (how to read logs).
  2. For Google surfaces, use URL Inspection in Search Console.
  3. Note the date of first verified recrawl as the start of the "after" period.

06Step 5: How do you analyse the result?

Illustrative, not real data:

GroupBeforeAfterChange
Test (n=160 each period)18%31%+13 points
Control (n=160 each period)20%23%+3 points
Difference-in-differences+10 points

Then ask:

  • Is the test group's change statistically meaningful? Run a two-proportion test, or compare Wilson intervals (guide).
  • Did the control move, signalling a background shift?
  • Does the qualitative evidence agree? For example, do answers now cite the changed pages or use their wording?

07What should you test first?

Choose changes that are cheap and plausibly important:

ChangeWhy it might matter
Answer-first introductionsClear, quotable statements
Comparison tablesStructured facts answers can draw on
Updated, dated factsReduces reliance on outdated sources
FAQ sections from real questionsMatches question-style prompts
Fixing crawler accessPrecondition for anything else

These are hypotheses, not proven tactics; that's the point of testing them.

08How do you avoid fooling yourself?

  • Pre-register the plan. Write hypothesis, metric, groups, duration and analysis method before starting.
  • Don't stop early on a good week. Decide duration in advance.
  • Don't add metrics after the fact. If citations didn't move, don't switch to mentions because they did.
  • Watch for contamination. Internal links from changed pages to control pages, or a press release about the topic, can affect controls.
  • Note model updates. If an engine changes model mid-test, consider extending or rerunning.

09What if you can't find a good control?

Small sites sometimes have no truly similar pages. Alternatives:

  • Staggered rollout: change half the test pages first and the rest a few weeks later. If visibility rises in step with each wave, the change is the more likely cause.
  • Competitor prompts as context: track how often rival brands appear in the same prompts; a general shift in who's recommended suggests an engine change.
  • Longer baselines: with more history, you can better judge whether a post-change jump is unusual.

None of these is as strong as a matched control, so state the design's limits in your write-up.

10How do you document results?

Keep a shared experiment log with: ID, hypothesis, dates, groups, n, results with ranges, conclusion (positive, negative, inconclusive) and next step. Over a year, the log becomes your team's most valuable GEO knowledge, because it's specific to your site and category.

11Common mistakes

  • Testing several changes at once.
  • No control group.
  • Measuring before crawlers revisit.
  • Too few observations to detect the effect.
  • Only sharing successful tests.

12From MentionX data: what makes GEO experiments noisyOriginal data

Method. Read-only aggregate queries over MentionX scans from 22 August to 29 October 2026 (UTC): 2,217 engine runs (2,154 completed) across 18 anonymised projects and six engines.

Three things in our data shape how you design tests.

First, answers vary. For 11 of 28 prompt-engine pairs we reran, results changed between runs. A single before-and-after check can't separate your change from that noise.

Second, engines behave very differently:

EngineBrand mentionedAnswers with no citations
Gemini47.3%50.7%
OpenAI42.9%30.2%
Perplexity31.5%4.1%
AI Mode31.3%12.6%
Copilot26.0%47.0%
DeepSeek24.0%99.0%

Some engines cite many sources (Perplexity averaged 9.56 citations per answer), so changes to a page can show up as citations. DeepSeek returned no citations in 99% of answers, so you can only measure mentions there.

Third, intent matters. Brand mention rates ranged from 5.7% for problem prompts to 100% for brand-specific prompts. Test and control prompts need the same intent mix, or the comparison isn't fair.

Our view: design for these limits up front. Experiments that ignore them tend to produce confident conclusions that don't repeat.

Limits. A convenience sample of mostly B2B software projects, run through engine APIs, which can behave differently from the consumer apps. Perplexity, Copilot and DeepSeek figures come from a single day (26 October 2026), so treat them as a snapshot, not a trend.

13Worked example: Ledgerline tests dated pricing summaries

Ledgerline is a fictional accounts-payable tool. Figures are illustrative.

Ledgerline has 12 integration pages with similar traffic and structure. Its hypothesis: adding a dated, plain-text pricing summary will raise how often engines cite these pages for pricing prompts.

The team splits the pages into two matched groups of six, by traffic and current citation rate. They track 24 pricing prompts on three engines, run three times a week, for four weeks of baseline. Then they add summaries to the test group only, request recrawling, and confirm indexing in Search Console.

After six weeks, the test group's citation rate has risen more than the control's, and a two-proportion test on the difference is significant. The team documents the result, including that one engine showed no change, then rolls the summaries out to the control group.

14How long should a test run?

Long enough for engines to see the change and for you to collect enough runs. In practice, that's usually four weeks of baseline and six to eight weeks after the change. Shorter tests risk measuring crawl delay rather than effect; longer ones risk being confounded by engine updates and seasonality.

15How do you deal with engine updates during a test?

Engines change their models and retrieval without notice. A control group is your main protection: if an update affects everything, it should affect both groups. Note any public announcements during the test and look for sudden shifts that hit test and control at the same time. If a large update lands mid-test, consider extending the test or starting again rather than reporting a muddied result.

16Which metrics should you choose?

Pick one primary metric that matches the hypothesis, and decide it before the test starts. For a change aimed at being cited, use citation rate for your pages. For a change aimed at being recommended, use brand mention rate or share of voice for the relevant prompts. Track a few secondary metrics, such as list position and referral traffic from AI assistants, but don't swap the primary metric afterwards because a secondary one looks better.

17How many prompts do you need?

Enough that your results can detect the effect you care about. As a rough guide, with about 100 observations per group per period, a rate near 50% has a 95% range of about ±10 points. If you expect a smaller effect, you need more prompts, more runs per prompt, or a longer test. Run the numbers before starting so you don't run a test that can't answer its own question.

18How do you match test and control groups?

Match on the things that affect AI visibility: current citation and mention rates, traffic, page type, topic and intent mix. Sort candidate pages by baseline rate and assign them alternately to each group, or randomly within pairs. Check the baselines are similar before you start. If one group starts much higher, results will be hard to interpret whatever happens.

19What should you do after a test?

Write up the hypothesis, method, result and decision, whether it worked or not. Roll out changes that worked, and consider retesting on a different page type. Keep a log of tests so later teams don't repeat them. Our view: a log of null results is valuable, because it stops teams spending months on tactics that don't move AI visibility for their site.

20Common questions

Can you A/B test a single page? Not cleanly, because engines see one version. Use groups of similar pages, or a before-and-after test with control pages.

Should you test on your most important pages? Start with pages that matter but aren't critical, so a bad result doesn't hurt core revenue. Move to key pages once a change has worked elsewhere.

What if engines don't cite sources? Measure brand mentions instead, and accept that effects on those engines will be slower and harder to attribute to a single page.

Can you run several tests at once? Yes, if they use different page groups and prompt sets that don't overlap.

21Checklist

  • Hypothesis with metric, prompt group and minimum effect
  • Matched test and control groups
  • Two to four weeks of baseline
  • Verified recrawl date recorded
  • Pre-registered duration and analysis
  • Results logged with ranges and conclusion

22Where to go next

Questions people ask

3 answers

Can you A/B test for AI visibility?

Not like a landing page. You can't split AI users, but you can compare changed pages with similar unchanged pages over time.

How long should a GEO test run?

Long enough for crawlers to revisit the pages and for enough prompt runs to build up, often several weeks or more.

What if results are inconclusive?

Record it honestly. An inconclusive test still tells you the effect, if any, was smaller than your test could detect.

Sources

  1. NIST/SEMATECH, Confidence intervals for proportionsitl.nist.gov
  2. Google, AI features and your websitedevelopers.google.com
Written and reviewed by Dipuck JonesHow we research and testReport an error

Keep learning

All Measurement →