GEO Experiment Design: Testing Changes

A GEO experiment compares AI visibility for pages or topics you changed against similar ones you didn't, before and after the change, so you can separate your effect from background shifts in AI answers. You can't randomly split users of ChatGPT, so the usual design is a controlled before-and-after comparison. Done carefully, it turns opinions about "what works in GEO" into evidence for your own site.

Key takeaways

• Write one hypothesis per test, with a metric and a minimum effect.
• Use control topics to account for engine-wide changes.
• Confirm crawlers have revisited changed pages before measuring.
• Report effects with ranges; publish inconclusive results internally too.

Why is GEO testing harder than CRO testing?

Those differences don't make testing impossible. They mean you need controls, enough observations and patience.

Step 1: How do you write a testable hypothesis?

Use the format: If we [change], then [metric] for [prompt group] will [direction] by at least [amount], because [reason].

Illustrative: "If we add a comparison table and a dated pricing summary to our five integration pages, then the recommendation rate for integration prompts in ChatGPT will increase by at least 10 points within eight weeks, because answers currently cite outdated third-party pages for pricing."

A good hypothesis names one change type. If you rewrite copy, add schema and build links at once, you won't know what worked.

Step 2: How do you choose test and control groups?

Pick pages or topics that are similar in traffic, age, type and current visibility. Change only the test group.

If both groups rise, an engine-wide change is the likely cause. If only the test group rises, your change is the more plausible explanation. This is a simple form of difference-in-differences: compare the change in test with the change in control.

Step 3: How do you set the baseline?

• Run the full prompt set for both groups for at least two to four weeks before the change.
• Record engine, model where visible, location and date for every run.
• Check the baseline is stable; if it swings wildly, you need more observations before testing.

Step 4: How do you know the change has been seen?

Publishing isn't the same as being processed. Before you start counting:

1. Check server logs for visits from relevant crawlers to the changed URLs (how to read logs).
2. For Google surfaces, use URL Inspection in Search Console.
3. Note the date of first verified recrawl as the start of the "after" period.

Step 5: How do you analyse the result?

Illustrative, not real data:

Then ask:

• Is the test group's change statistically meaningful? Run a two-proportion test, or compare Wilson intervals (guide).
• Did the control move, signalling a background shift?
• Does the qualitative evidence agree? For example, do answers now cite the changed pages or use their wording?

What should you test first?

Choose changes that are cheap and plausibly important:

These are hypotheses, not proven tactics; that's the point of testing them.

How do you avoid fooling yourself?

• Pre-register the plan. Write hypothesis, metric, groups, duration and analysis method before starting.
• Don't stop early on a good week. Decide duration in advance.
• Don't add metrics after the fact. If citations didn't move, don't switch to mentions because they did.
• Watch for contamination. Internal links from changed pages to control pages, or a press release about the topic, can affect controls.
• Note model updates. If an engine changes model mid-test, consider extending or rerunning.

What if you can't find a good control?

Small sites sometimes have no truly similar pages. Alternatives:

• Staggered rollout: change half the test pages first and the rest a few weeks later. If visibility rises in step with each wave, the change is the more likely cause.
• Competitor prompts as context: track how often rival brands appear in the same prompts; a general shift in who's recommended suggests an engine change.
• Longer baselines: with more history, you can better judge whether a post-change jump is unusual.

None of these is as strong as a matched control, so state the design's limits in your write-up.

How do you document results?

Keep a shared experiment log with: ID, hypothesis, dates, groups, n, results with ranges, conclusion (positive, negative, inconclusive) and next step. Over a year, the log becomes your team's most valuable GEO knowledge, because it's specific to your site and category.

Common mistakes

• Testing several changes at once.
• No control group.
• Measuring before crawlers revisit.
• Too few observations to detect the effect.
• Only sharing successful tests.

Checklist

• [ ] Hypothesis with metric, prompt group and minimum effect
• [ ] Matched test and control groups
• [ ] Two to four weeks of baseline
• [ ] Verified recrawl date recorded
• [ ] Pre-registered duration and analysis
• [ ] Results logged with ranges and conclusion

Where to go next

• Statistical confidence for AI visibility tracking
• AI visibility KPIs for marketing teams

Sources

• NIST/SEMATECH, Confidence intervals for proportions
• Google, AI features and your website