A Trustworthy Experiment Checklist: What 20,000 Annual Tests at Microsoft Taught About A/B Testing

A significant number is not a trustworthy one

The reason most reported A/B test wins do not survive is not bad luck. It is that the number was believed before it was checked. A conversion tool will happily display a green "95% confidence" figure the moment a variant edges ahead, and that figure feels like a verdict. It is not. The confidence statement is only valid if the experiment behind it was designed and read according to rules set before any data arrived, and in practice those rules are the exception rather than the norm.

The definitive statement of what those rules are comes from practice, not theory. In Trustworthy Online Controlled Experiments, Ron Kohavi, Diane Tang and Ya Xu, who led experimentation at Microsoft and adjacent organizations, distilled the lessons of more than 20,000 controlled experiments run each year at Microsoft into a working discipline. Their central claim is deceptively plain: trust is not a property of the p-value, it is a property of the process. A test earns belief only after a defined set of failure modes has been ruled out. The checklist below is a plain-language reading of that canon.

The stakes are not academic. When a business ships a change because a test "won", it is committing real money and real customer experience to a claim. If the claim was an artifact of a broken split or an early peek, the business has not just wasted the test; it has baked a false belief into the site and will defend it for years. Measurement discipline, not creativity, is the scarce resource here.

Twyman's law: distrust the result that thrills you

The first check runs before any statistics: the more surprising a result is, the more likely it is an error. This heuristic, which the trustworthy-experiment canon names Twyman's law, inverts the usual instinct. A dramatic lift is not cause for celebration; it is cause for suspicion. Genuinely large effects from small changes are rare, so an eye-catching number is, on the base rates alone, more often a logging bug, a misfired event, a bot surge, or a broken split than a breakthrough.

The discipline of doubting your own good news has a hard evidential backbone. In large-scale randomized experiments at eBay, Blake, Nosko and Tadelis found that observational, attribution-style estimates of paid-search return were systematically inflated relative to the true causal lift, and that brand-keyword ads showed no measurable incremental short-term benefit once the effect was measured with a real experiment rather than inferred from a dashboard. The comfortable number was the wrong one. Twyman's law is the habit of assuming the comfortable number is wrong until it has survived scrutiny.

Operationally, this means a surprising result triggers investigation, not a launch. You re-check the instrumentation, confirm the metric fired once on the right action, look for an external event during the test window, and only then ask whether the effect might be real. A result that cannot explain its own size does not get shipped.

Sample ratio mismatch: the check that voids the test

The single most important mechanical check is for sample ratio mismatch, or SRM. If you assigned visitors to treatment and control in a 50/50 split, the counts at the end should be close to 50/50. When the observed split deviates from the assigned ratio by more than chance can explain, the randomization itself is compromised, and every downstream metric is untrustworthy, no matter how clean or significant the lift looks.

This is not a rare edge case. Analyzing the failure mode across production systems, Fabijan and colleagues, alongside Kohavi's own trustworthy-experiment work, found that roughly 6 percent of online controlled experiments trip an SRM check. When they do, the correct response is not to interpret the result cautiously. It is to discard it entirely and find the cause, because an SRM means the two groups being compared were never actually comparable.

The peeking problem: when you look changes what you can conclude

The second mechanical failure is peeking: watching a running test and stopping the moment it crosses significance. It feels responsible, and it is one of the most reliable ways to manufacture a false win. Classical significance testing assumes a single analysis at a predetermined sample size. Every additional look at the data is another chance for random noise to cross the threshold, so repeated peeking inflates the false-positive rate far beyond the nominal 5 percent.

The magnitude is documented. Johari, Koomen, Pekelis and Walsh, analyzing a large base of real experiments, showed that continuously monitoring a fixed-horizon test against a nominal p<0.05 threshold pushes the realized false-positive rate well above 5 percent, with follow-on analyses putting it above 40 percent under aggressive peeking. In other words, a team that checks daily and stops on the first green reading can be wrong close to half the time while believing it is wrong one time in twenty.

There are two legitimate fixes, and one illegitimate one. The illegitimate fix is to keep peeking but "wait a bit longer" on gut feel, which is just peeking with extra steps. The legitimate fixes are either to fix the sample size and reading point in advance and look only once at the end, or to adopt an always-valid sequential method designed so that continuous monitoring is statistically sound. The checklist requires one or the other to be chosen before launch, never decided while the numbers are moving.

Novelty and primacy: the effect that fades

Even a correctly split, correctly read test can mislead over a short window because of two time-dependent effects the canon names explicitly. A novelty effect is a temporary bump caused by regular users noticing something new and clicking out of curiosity; it decays as the novelty wears off. A primacy effect is the opposite: experienced users are briefly slowed or annoyed by a change they were used to, so the treatment looks worse at first and recovers as they adapt.

Both mean the first days of a test can point the wrong way. A change that "wins" for a week on novelty can be flat or negative once the curiosity fades, and a change that "loses" early on primacy can be the right long-term call. The defense is to run tests over a period long enough for these transients to settle, typically at least one full week and usually in whole weeks so that day-of-week patterns do not skew the read, and to inspect the effect over time rather than trusting a single cumulative number. A flat trend line that started high is a novelty artifact wearing a winner's costume.

Network interference: when treatment leaks into control

The last structural check asks whether the two groups are truly independent. Standard A/B analysis assumes that what you do to a treated user has no effect on a control user. Network interference is the violation of that assumption: treatment leaks across the boundary, so the control group is no longer a clean counterfactual and the measured difference understates or overstates the true effect.

For most single-site conversion tests on unrelated visitors, this assumption holds and the check is quickly cleared. It matters most where users interact, such as marketplaces, social features, referral loops, or any system with shared inventory or shared supply. If a treated seller's behavior changes the experience of a control buyer, the arms are entangled. The canon flags this as a prerequisite to reason about, not a box to tick blindly: you confirm the unit of randomization is genuinely isolated from the units in the other arm, and if it is not, you change the design, for example by randomizing at the level of a cluster rather than an individual.

The pre-launch checklist

Collapsed into a single pass, the trustworthy-experiment canon becomes a short list a small team can run before it believes any A/B result. Each item corresponds to a documented failure mode above, and each has a binary answer. If any answer is no, the result is not yet trustworthy.

• Hypothesis and metric fixed first: one written hypothesis and one primary metric named before launch, so success cannot be redefined after the fact.
• Sample size and reading point set in advance: the minimum detectable effect, the required sample size, and a single end-of-test reading point are calculated before any traffic is split, or an always-valid sequential method is chosen instead.
• Sample ratio mismatch cleared: the observed group split matches the assigned ratio within chance (chi-squared test); if it fails, the test is void and the cause is found before anything is read.
• No peeking, or valid peeking only: the result is read once at the planned endpoint, unless an always-valid method licenses continuous monitoring.
• Run long enough for novelty and primacy to settle: at least one full week, usually in whole weeks, with the effect inspected over time rather than as a single cumulative figure.
• Network interference ruled out: the unit of randomization is genuinely isolated from the other arm, or the design is changed until it is.
• Twyman's law applied: any surprising result is investigated as a likely error before it is treated as a discovery.
• Effect reported with its uncertainty: the read-out states the effect and its confidence interval, and a flat or losing result is reported as plainly as a win.

What this means when your traffic is small

One caveat separates this canon from its origin. Microsoft, eBay and Optimizely's customers generate the traffic to run thousands of well-powered tests. A single med spa, law practice or home-services site does not. At low volume, the same checklist still governs whether a result is trustworthy, but it changes what is testable: run lengths stretch, the smallest detectable effect grows, and some tests simply cannot reach a trustworthy conclusion in a reasonable window. Pretending otherwise, by stopping early or ignoring an SRM, is how small businesses accumulate a folder of "wins" they cannot rely on.

The correct response for a small operator is not to abandon experimentation but to be selective and disciplined: test only changes big enough to detect at your traffic, on the pages that carry the most revenue, and honor every check even when the sample is thin. Where a page cannot support a clean test, the right move is to say so and to fix known friction on evidence rather than to run a test that cannot answer the question. The discipline is the same as it is at scale; the scarcity is just more acute.

That is the whole point of the checklist. It is the condition under which acting is justified. A test that passes every item gives you a result you can spend money on. A test that fails one gives you a number, and a number that has not cleared these checks is not yet knowledge.