Conversion Science · established evidence
The Peeking Problem: Why Checking Your A/B Test Dashboard Early Lies to You
A/B test peeking, checking the dashboard while a test runs and stopping the moment the result looks significant, is the most common way experimentation teams fool themselves. A standard fixed-horizon test is designed for a single look at a pre-set sample size, and its p<0.05 threshold delivers the promised 5 percent false-positive rate only if you look exactly once. Every additional glance is another chance for random noise to cross the line, so continuous monitoring quietly inflates the real false-positive rate well above the 5 percent you believe you are holding. In 2017, four researchers analyzing Optimizely's customer base measured this effect and published the fix. This piece explains why early stopping lies to you, what the study actually showed, and why only always-valid sequential inference lets you watch a test in real time without corrupting the result.
A dashboard that updates is an invitation to stop early
Modern experimentation tools show a running result that refreshes by the hour. The p-value ticks down, a confidence interval narrows, and at some point the tool paints the variant green. The natural human move is to stop there and ship the winner. That move is the error.
A conventional A/B test is a fixed-horizon procedure. You decide the sample size before it starts, you collect exactly that much data, and you read the result once at the end. The statistics that justify the p<0.05 cutoff assume that single reading. The moment you watch the test unfold and reserve the right to stop when it looks good, you are no longer running the test the math describes.
This is not a matter of impatience or weak discipline. It is a structural property of the significance test itself, and it holds no matter how careful the person watching the dashboard is.
Why early stopping in an ab test inflates false positives
A p-value below 0.05 is a promise about a long run: if the variant truly does nothing, a single fixed-horizon test will still cross the 0.05 line by chance about 5 percent of the time. That 5 percent is the false-positive rate you are agreeing to when you set the threshold.
Peeking breaks the promise by turning one test into many. Each time you look and apply the same 0.05 cutoff, you give random noise a fresh opportunity to trip it. A result that is drifting around zero will, given enough looks, eventually wander across the significance line for a moment purely by chance, and an eager watcher stops exactly then.
One threshold, many chances to cross it
The underlying issue is the multiple-comparisons problem wearing a different coat. Testing one hypothesis at 0.05 controls the error at 5 percent. Testing the same hypothesis again and again as data trickles in, and accepting the first crossing, is closer to running many tests and keeping only the one that passed. The more often you look, the higher the chance that at least one look shows a false win.
This is why a test can look decisively positive on Tuesday, flat on Thursday, and positive again the following week. The signal was never stable. The dashboard simply caught it mid-wander, and a stop-when-green rule cashes in on the noise.
What the Optimizely study actually measured
The definitive treatment of this is Johari, Koomen, Pekelis and Walsh, "Peeking at A/B Tests: Why It Matters, and What to Do About It," published at KDD 2017 and grounded in analysis of Optimizely's own customer base. The authors show that continuously monitoring a fixed-horizon test against a nominal p<0.05 threshold pushes the realized false-positive rate well above 5 percent.
The size of the inflation depends on how often and how aggressively the test is watched, and commonly cited analyses following this work put the realized false-positive rate above 40 percent under aggressive peeking. Read that plainly: a nominal 5 percent test, watched hard and stopped early, can be wrong close to half the time it declares a winner. The honesty of the reported number collapses long before anyone notices, because the failure is invisible in the dashboard that produced it.
The important detail is that these were not lab simulations of a hypothetical. The behavior was observed in a live commercial experimentation platform used by real teams shipping real changes, which is why the finding carries weight beyond a textbook caution.
False positive ab testing is a statistics problem, not a willpower problem
It is tempting to treat peeking as a bad habit to be scolded out of analysts. That framing misses the point. The fixed-horizon significance test is simply the wrong instrument for a data stream you intend to watch. Asking people to look at a live dashboard and not react is asking them to ignore the tool's entire reason for updating.
The clean way to see it: a fixed-horizon p-value answers the question "if I had run this to the planned sample size and looked once, how surprising is this result?" It cannot answer "given that I have been watching and chose to stop now, how surprising is this result?" Those are different questions, and using the answer to the first as if it answered the second is the whole error.
Always valid inference: the fix built for continuous monitoring
The Johari et al. paper does not just diagnose the problem, it supplies the remedy: always-valid inference. Instead of a p-value that is only trustworthy at one pre-committed sample size, always-valid methods produce always-valid p-values and confidence sequences that remain statistically honest at every point in time, no matter when or how often you look.
The mechanism is sequential testing rather than fixed-horizon testing. A sequential procedure, the paper uses a mixture sequential probability ratio test, spends its error budget across the whole monitoring period rather than assuming a single look. The practical payoff is exactly what practitioners wanted all along: you can watch the test live, stop as soon as the evidence is genuinely decisive, and still hold the false-positive rate you promised. The fix is a better test, not a rule against looking.
What does not fix it is the folk remedy of quietly extending the test whenever the result is not yet significant, then reading it again. That is peeking by another name, and it inflates the error the same way. Ad hoc sample-size padding is not sequential inference.
When to stop an ab test: the two honest options
There are exactly two defensible ways to run an experiment you can trust, and both are decided before the test starts, not in the heat of a promising dashboard.
- Fix the horizon and look once. Set the minimum effect worth detecting, calculate the required sample size, run to that number, and read the result a single time at the end. Simple, dependable, and the classical guarantee holds exactly.
- Use a sequential method built for monitoring. Adopt an always-valid or group-sequential design up front, and then you may watch continuously and stop the moment the evidence is decisive, because the method was constructed to permit that.
- What is never valid is the third path most teams actually take by accident: run a fixed-horizon test, watch it every day, and stop whenever it first looks good. That combination gets the worst of both worlds, the convenience of watching and the false confidence of a guarantee that no longer applies.
Peeking is one of several ways a trusted test quietly lies
Peeking rarely travels alone. The same practitioner literature that names it also documents other silent invalidators, which is why serious experimentation is a checklist, not a single p-value.
One of the most common is sample ratio mismatch: when the actual split between treatment and control deviates from the assigned ratio at high significance, the result cannot be trusted regardless of the observed lift. Fabijan and colleagues, analyzing Microsoft's experimentation platform, found roughly 6 percent of production experiments trip such a check. Ron Kohavi, Diane Tang and Ya Xu's book, built on more than 20,000 annual experiments at Microsoft, gathers these prerequisites and offers a governing heuristic worth taping to the wall, Twyman's law: a result that looks surprising or too good is usually a measurement error before it is a discovery.
The through line is that a green dashboard is a claim, not a proof. Peeking, sample ratio mismatch, novelty effects and interference each turn a confident-looking number into an artifact. Treating the number as evidence, rather than as a hypothesis about your evidence, is the mistake the whole discipline exists to prevent.
How to tell whether your reported wins survived the peek
You do not need a statistics degree to pressure-test a past result. Ask three questions of any A/B test that was called a winner. Was the sample size fixed in advance, or did the test simply run until it looked good? Was the result read once at a pre-set endpoint, or was it stopped on the day it first crossed significance? And if it was monitored continuously, was the analysis an always-valid or sequential method, or an ordinary fixed-horizon p-value applied to a moving picture?
If a "win" was watched daily and stopped on first significance under a plain p<0.05 test, the honest position is that you do not yet know whether it was real. That is not a reason to distrust experimentation. It is the reason to run it properly, because a test done right is still the most reliable way to know what actually moves your buyers.
The evidence
Key findings, with their sources
-
Continuously monitoring a fixed-horizon test against a nominal p<0.05 threshold pushes the realized false-positive rate well above the promised 5 percent; commonly cited analyses of aggressive peeking put it above 40 percent.
established Johari, Koomen, Pekelis & Walsh, "Peeking at A/B Tests: Why It Matters, and What to Do About It", KDD '17, doi:10.1145/3097983.3097992 (analysis of Optimizely's customer base).
-
The remedy is always-valid sequential inference, always-valid p-values and confidence sequences that stay honest at every look, not ad hoc extension of the sample size.
established Johari, Koomen, Pekelis & Walsh, "Peeking at A/B Tests", KDD '17, doi:10.1145/3097983.3097992.
-
Roughly 6 percent of production experiments trip a sample-ratio-mismatch check, meaning the treatment/control split deviates from the assigned ratio at high significance and the result cannot be trusted regardless of the observed lift.
established Fabijan et al., "Diagnosing Sample Ratio Mismatch in Online Controlled Experiments", KDD '19, doi:10.1145/3292500.3330722; drawing on Ron Kohavi's Microsoft experimentation work.
-
A trustworthy online experiment requires a defined set of prerequisite checks, not just a p-value, governed by Twyman's law: a surprising result is usually an error before it is a discovery.
established Kohavi, Tang & Xu, "Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing", Cambridge University Press, 2020, ISBN 9781108724265 (built on more than 20,000 annual experiments at Microsoft).
Calibration
What is proven, what is promising, what is unproven
| Evidence tier | Tactics | What the evidence says |
|---|---|---|
| established | Peeking inflates false positives; a fixed-horizon p-value is only valid at a single pre-set look | Johari et al., KDD '17 (Optimizely); Kohavi, Tang & Xu, 2020 |
| established | Always-valid and sequential inference permit continuous monitoring without corrupting the error rate | Johari et al., KDD '17 (always-valid p-values, confidence sequences, mixture SPRT) |
| emerging | The exact realized false-positive rate under aggressive peeking (the commonly cited 40 percent-plus figure) | Follow-on analyses of Johari et al.; the precise number depends on how often and how hard a test is watched |
Reference
Glossary
- Peeking
- Repeatedly checking a running A/B test and stopping as soon as the result looks statistically significant, instead of reading it once at a pre-set endpoint.
- Fixed-horizon test
- The classical A/B design: sample size is fixed in advance, data is collected to that size, and the result is read exactly once. The p<0.05 guarantee assumes this single look.
- False positive (Type I error)
- Declaring a variant a winner when it truly does nothing. A p<0.05 threshold is meant to hold this rate at 5 percent for one fixed-horizon look.
- Always-valid inference
- Statistical methods that produce p-values and confidence sequences which remain valid at every point in time, so a test can be monitored continuously and stopped early without inflating the error rate.
- Sequential testing
- A test designed to be evaluated repeatedly as data arrives, spending its error budget across the whole monitoring period rather than assuming a single endpoint. The mixture sequential probability ratio test is one such method.
- Sample ratio mismatch
- When the actual split between treatment and control deviates from the assigned ratio at high significance, signaling a flaw that invalidates the result no matter how large the observed lift.
Straight answers
Frequently asked questions
What is "peeking" in A/B testing?
Peeking is checking a running test on a live dashboard and stopping it the moment the result crosses your significance threshold. It feels efficient, but a standard test's p<0.05 guarantee only holds if you look once at a pre-set sample size. Every extra look gives random noise another chance to trip the threshold, so stopping on first significance inflates your false-positive rate.
Does checking my test early really change the result?
Yes, and not because the data changes, but because the guarantee does. A fixed-horizon p-value answers "how surprising is this if I looked once at the planned sample size?" It cannot answer "how surprising is this given that I have been watching and chose to stop now?" Using the first answer as if it were the second is the error, and it makes wins look more certain than they are.
How high can the false-positive rate actually go?
Higher than most teams expect. Johari, Koomen, Pekelis and Walsh, analyzing Optimizely's customer base at KDD 2017, showed continuous monitoring against a nominal 5 percent threshold pushes the real false-positive rate well above 5 percent, and commonly cited analyses of aggressive peeking put it above 40 percent. The precise figure depends on how often and how hard the test is watched.
Can I ever look at a test before it finishes?
Yes, if you use the right method. Always-valid and sequential designs are built to be monitored continuously and stopped as soon as the evidence is genuinely decisive, while still holding the false-positive rate you promised. What breaks the math is watching an ordinary fixed-horizon test and stopping on first significance, or quietly extending it and reading again.
What is always-valid inference?
It is the fix proposed in the same research that named the problem: statistical methods whose p-values and confidence intervals stay honest at every moment, not just at one pre-committed sample size. That lets you watch a test live and act early without corrupting the result, which is exactly what a live dashboard tempts everyone to do.
Provenance
Sources
- Johari, R., Koomen, P., Pekelis, L. & Walsh, D., "Peeking at A/B Tests: Why It Matters, and What to Do About It", KDD '17, doi:10.1145/3097983.3097992 (established)doi.org
- Kohavi, R., Tang, D. & Xu, Y., "Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing", Cambridge University Press, 2020, ISBN 9781108724265 (established)cambridge.org
- Fabijan, A., Gupchup, J., Gupta, S., Omhover, J., Qin, W., Vermeer, L. & Dmitriev, P., "Diagnosing Sample Ratio Mismatch in Online Controlled Experiments", KDD '19, doi:10.1145/3292500.3330722 (established)doi.org
Every figure above is attributed to a real, dated source and tagged with its evidence tier. Where a claim could not be verified to a primary source, it is not stated as fact.