Most A/B tests die not from a bad hypothesis but from a bad finish line. Someone ships a variant, watches the dashboard for four days, sees the line tick up, and declares victory. Then the effect evaporates on the next cohort, and nobody can explain why. The numbers were real; the reading of them was not.
There are two ways to run an experiment. The intuitive one: launch, watch, stop when it looks good. The statistically correct one: fix the metric and the sample size before launch, compute power in advance, and do not peek. The gap between them is not academic — it is the difference between a decision and a coin flip dressed as a decision.
An LLM is genuinely useful here, but only if you prompt it as a statistician, not as a hype machine. Ask "is my A/B test winning?" and it will happily agree with you. Ask it to compute the sample size, check the confidence interval, and audit the design for peeking and metric drift — and it becomes the reviewer your growth team does not have.
Pick your lane: traffic volume, error budget, and the price of a false win
The choice between approaches is not about sophistication. It is about three inputs you can actually estimate.
| Factor | Intuitive approach | Statistically correct approach |
|---|---|---|
| Minimum traffic | Tiny, any volume | Needs enough users to detect a realistic effect |
| Tolerance for error | High — you accept false positives | You set alpha and beta explicitly |
| Cost of a wrong call | Low — cheap to revert | High — pricing, checkout, retention |
| Time to decision | Days | Weeks, fixed in advance |
A rough rule: if a false positive would cost you a refund wave, a pricing mistake, or a churned cohort, you cannot afford the intuitive lane. If you are testing button copy on a page with 200 visits a day, a formal test will never reach power — and pretending otherwise is worse than shipping on judgment.
Prompt 1: sample size before you touch the code
The single most expensive mistake is launching a test that can never reach significance for the effect size you care about. This prompt forces the arithmetic up front.
Prompt:
You are a senior experimentation statistician. I am planning an A/B test and need the required sample size BEFORE launch.
Inputs:
- Baseline conversion rate: 4.2%
- Minimum detectable effect I care about: a relative lift of 10% (absolute 0.42 pp)
- Significance level alpha: 0.05 (two-sided)
- Power: 0.80
- Daily eligible traffic split 50/50: 6,000 users per arm
- Primary metric: purchase conversion
Tasks:
1. Compute the sample size per arm using the standard two-proportion formula. Show the formula and every substituted number.
2. Convert it into calendar days given the traffic split, and round up.
3. Recompute for a 5% relative MDE and tell me how the duration changes.
4. State plainly whether a 10% MDE is realistic here, and if not, what the smallest detectable effect is for a 14-day test.
Do not give a range. Give one number per scenario and show the arithmetic so I can verify it.
Why it works: the demand to show arithmetic is the guardrail. An LLM that has to substitute numbers into the two-proportion formula is far less likely to hand you a confident-sounding guess. For a baseline of 4.2% and a 10% relative lift, the per-arm requirement lands in the low tens of thousands — meaning a 6,000-per-arm daily flow needs roughly a week or more, not the two days most teams budget.
Prompt 2: read the result without fooling yourself
Once data is in, the temptation is to ask "did we win?" Ask instead for the interval and the caveats.
Prompt:
Act as a skeptical reviewer, not a cheerleader. Here are my results:
- Control: 1,842 conversions / 44,100 users
- Variant: 1,961 conversions / 44,050 users
- Test ran 9 days, planned 14. We stopped early because it looked good.
Do:
1. Compute conversion rates and the absolute and relative lift.
2. Compute a 95% confidence interval for the difference in proportions and give the p-value for a two-sided test.
3. Tell me explicitly whether stopping early invalidates the result, and why.
4. List what I would need to check before trusting this: sample ratio mismatch, novelty effect, segment consistency.
Be blunt. If the interval crosses zero, say so in the first sentence.
The payoff is in point 3. Early stopping inflates the false-positive rate — a test peeked at repeatedly and stopped at the first crossing can produce a "significant" result far more often than alpha suggests. The model will flag it; a dashboard will not.
Prompt 3: the design audit that catches podvokhi
Before you trust any result, run the design through an adversarial check. This is where an LLM earns its keep — it reads your experiment spec the way a reviewer would.
Prompt:
Review this experiment design and list every flaw that could produce a false positive or a misleading readout. For each flaw: name it, explain the mechanism, and give the fix.
Design:
- Hypothesis: new pricing page increases trial signups
- Primary metric: trial signups
- Secondary metrics: 11 others, we will report whichever moves
- Randomization: by user ID, but returning visitors keep their old assignment
- Duration: until we see significance
- Segments: we will slice by country, device, and plan after the fact
Rank flaws by how badly they corrupt the conclusion.
Expect the model to catch multiple comparisons (11 secondary metrics guarantee a "winner" by chance), assignment drift for returning users, and the open-ended duration. These are the podvokhi — the traps that make a losing variant look like a winner.
Prompt 4: the pre-launch checklist
Turn the whole thing into a gate. Paste this before every experiment and require every answer to be yes.
Prompt:
Generate a pre-launch checklist for my A/B test as a numbered list, each item a yes/no question with a one-line reason. Cover: single primary metric defined, sample size computed, duration fixed, no peeking policy, sample ratio mismatch monitor, guardrail metrics, and a written decision rule for what happens on a win, a loss, and a flat result. End with the three items most often skipped and why they matter.
The decision rule is the part teams skip. "If the interval excludes zero, ship; if it includes zero, revert; if flat, keep control and log the hypothesis for a bigger test" — writing that down before launch removes the negotiation that happens when the result is inconvenient.
Where the approach breaks
An LLM is a calculator with judgment, not an oracle. It will not know your traffic seasonality, your bot filtering, or whether your conversion event double-fires. Feed it clean inputs and verify the arithmetic; treat its statistical reasoning as a second opinion, not a verdict.
If your pipeline feeding the experiment is itself unreliable, no prompt saves you — start with observability in production so the numbers you argue about are real. And when you finally visualize the result for stakeholders, remember that most charts fail — a confidence interval drawn honestly beats a bar chart that implies certainty you do not have.
Fix your metric and sample size before launch, let the LLM audit the design, and refuse to call a winner until the interval says so. That discipline is the whole game — and it is a habit you can build this week, one prompt at a time.