Course lesson · Beginner
Your First Marketing Experiment
A marketing experiment tests one change against a clear hypothesis, using one metric, a baseline, a control and a date fixed in advance, so you learn whether the change worked rather than guessing. AI visibility and traffic are noisy, so simple, disciplined experiments are the most reliable way to find what actually helps your brand.
In this lesson
- The four parts of an experiment
- Writing a good hypothesis
- Controls and timing
- Reading results without fooling yourself
- Exercise and quiz
What are the four parts?
| Part | Question it answers | Example (illustrative) |
|---|---|---|
| Hypothesis | What do you expect, and why? | "Adding answer-first openings to 5 guides will raise their AI citation rate" |
| Metric | How will you measure it? | Citation rate across 20 prompts × 3 engines |
| Control | What will you compare with? | 5 similar guides left unchanged |
| Date | When will you decide? | Six weeks after publishing |
How do you write a good hypothesis?
Use: If we [change], then [metric] will [direction], because [reason].
- Specific: one change, not five at once
- Measurable: a metric you can actually collect
- Reasoned: based on evidence or guidance, so you learn something either way
Why do you need a control?
Many things move metrics: model updates, seasonality, press coverage, competitor changes. If your changed pages and your control pages rise by the same amount, your change probably wasn't the cause. Controls are the cheapest way to avoid false wins.
How should you time it?
- Measure a baseline first, with repeat runs, because AI answers vary.
- Fix the decision date in advance. Checking daily and stopping when the numbers look good is a classic way to fool yourself.
- Allow time for recrawling and system updates; several weeks is common.
How do you read the results?
- Compare the change versus baseline for both test and control groups.
- Report ranges for rates, such as confidence intervals (NIST), not just single numbers.
- Note confounders, such as a launch or PR spike.
- Decide: roll out, iterate or stop.
- Log the result, including null results.
Which experiments suit beginners?
| Experiment (illustrative) | Change | Metric |
|---|---|---|
| Answer-first openings | Rewrite the first paragraph of 5 pages | Citation rate for those pages' prompts |
| FAQ additions | Add 5 follow-ups to 5 pages | Mention rate for follow-up prompts |
| Positioning consistency | Align descriptions on 10 profiles | Share of answers that describe you correctly |
| Comparison table | Add a table to 5 pages | Citation rate for comparison prompts |
| Call to action | Clearer CTA on AI-referred landing pages | Key event rate from AI referrals |
Start with the experiment that's cheapest to run and closest to revenue.
What are the most common mistakes?
- Changing several things at once, so you can't tell what worked
- No baseline, so there's nothing to compare against
- Too few prompts or runs, so results are mostly noise
- Moving the goalposts: switching metrics after seeing the results
- Not writing anything down, so the lesson is lost
How do you write up the result?
Use one short template for every experiment:
| Field | Example (illustrative) |
|---|---|
| Hypothesis | Answer-first openings raise citation rate |
| Change and date | 5 pages rewritten, 3 March |
| Prompts and runs | 20 prompts × 5 runs × 3 engines |
| Baseline | 12% (8–17%) |
| Result | 21% (16–27%) |
| Control | 13% → 14% |
| Decision | Roll out to 20 more pages |
Share the write-up with your team even when the test fails. A failed test that's documented still saves the next person time.
When should you stop an experiment early?
Stop it only if the change causes harm: wrong facts appear in answers, conversions fall sharply, or legal raises a problem. Otherwise run it for the full planned period. Stopping as soon as the numbers look good inflates the apparent result.
A quick example
Illustrative. A team adds a clear comparison table to five product pages (test) and leaves five similar pages alone (control). Baseline: test pages cited in 12% of runs, control 11%. After six weeks: test 21%, control 12%. The team rolls the table format out to ten more pages and repeats the measurement.
Experiment log template
| Field | Entry |
|---|---|
| Hypothesis | |
| Change | |
| Pages / prompts | |
| Primary metric | |
| Control | |
| Baseline (date, value) | |
| Decision date | |
| Result (value, range) | |
| Decision | |
| Notes / confounders |
Experiment checklist
- [ ] One change, one hypothesis
- [ ] One primary metric
- [ ] Baseline with repeat runs
- [ ] Control group
- [ ] Decision date fixed
- [ ] Ranges reported
- [ ] Result logged
Exercise
Design a before/after test: hypothesis, metric, date; log it.
- Pick one change you learned in this course, such as answer-first openings or clearer positioning.
- Fill in the experiment log template.
- Record your baseline this week.
- Put the decision date in your calendar.
Quiz
- What should a hypothesis include?
a) A change, an expected result and a reason b) Only a goal c) A budget Answer: a
- Why use a control group?
a) To see whether something else caused the change b) To double the work c) To hide results Answer: a
- When should you fix the decision date?
a) Before you start b) When results look good c) Never Answer: a
- What should you do with a null result?
a) Log it b) Ignore it c) Change the data Answer: a
- How many changes should one simple experiment test?
a) One b) Five c) As many as possible Answer: a
Next: You've finished AI-Era Marketing Foundations. Continue with GEO 101, Lesson 1: What GEO is and why it matters
Sources
- NIST/SEMATECH, Confidence intervals for proportions
Questions
Do I need statistics knowledge to run an experiment?
Only the basics: a clear metric, a baseline, a control and an understanding that small samples are noisy.
What's a control?
Something similar that you don't change, so you can tell whether your change or something else caused the result.
What if the experiment shows nothing?
Log it. A clear 'no measurable effect' stops you investing in tactics that don't work.