Why Most A/B Tests Are Wrong Before They Start: Sample Ratio Mismatch, Explained

A winning result can already be broken

Most conversations about A/B testing focus on the result: which variant won, by how much, and at what confidence. Sample Ratio Mismatch matters because it sits underneath all of that. It is a property of how the experiment was run, and when it is present, the result on top of it is not a smaller win or a noisier win. It is an uninterpretable one.

The logic is simple. An online controlled experiment works by randomly assigning comparable visitors to control and treatment, so that the only systematic difference between the groups is the change being tested. If the split that actually occurs departs from the split that was designed, that assumption of comparability is gone. Something selected who ended up in which bucket, and whatever did the selecting is now tangled up with your metric. You can no longer attribute the difference in conversion to the change you made, because you can no longer rule out that the two audiences were simply different people.

Kohavi, Tang and Xu, whose Trustworthy Online Controlled Experiments distils experimentation practice built on more than 20,000 experiments a year at Microsoft, place the SRM check among the small set of validity checks that must pass before a result is believed at all. A significant mismatch is treated as a signal to invalidate and investigate, not to discount slightly.

What sample ratio mismatch actually is

Define the terms plainly. The designed ratio is the allocation you chose when you set the test up, for example 1:1 between two variants. The observed ratio is the count of users, sessions, or whatever unit you randomized on, that actually landed in each variant when the test ran. Sample Ratio Mismatch is a statistically significant gap between the two.

Because random assignment naturally produces small fluctuations, you do not flag every imperfect split. A 50.2 / 49.8 outcome on a large sample is expected noise. The point of a formal check is to separate ordinary sampling variation from a deviation too large to be chance. When the observed counts are far enough from the designed ratio that random variation is an implausible explanation, the experiment has an SRM, and the burden shifts to finding the mechanism that caused it.

The SRM check: one test that decides whether to trust the rest

The check itself is a goodness-of-fit test. You take the counts that actually arrived in each variant and compare them against the counts the designed ratio predicts, using a chi-squared goodness-of-fit test, and read the p-value. A very small p-value means the split you got would almost never happen by chance if assignment were truly random and unbiased, which is the signature of a mismatch.

Two things make this check unusual, and worth understanding before you rely on it.

Why roughly 6% of production tests fail this check

The reason SRM deserves a dedicated article is that it is common, not rare. Microsoft experimentation work reported that on the order of 6% of controlled experiments exhibit a sample ratio mismatch. In practice that means a meaningful share of the tests a team runs, and celebrates, are resting on a split that a formal check would reject.

That figure reframes the usual anxiety about A/B testing. The worry is normally about the outcome, whether a lift is real or noise. SRM shows that a portion of the problem is upstream of the outcome entirely: the experiment was compromised at the assignment layer, and no amount of care in reading the result recovers it. This is a large part of the answer to why A/B tests fail. Many of them fail before the first visitor is counted, in the plumbing that decides who sees what.

Where the mismatch comes from

Fabijan and colleagues, in "Diagnosing Sample Ratio Mismatch in Online Controlled Experiments," give practitioners a taxonomy of causes and rules of thumb for tracking them down. The value of their work is that SRM is diagnosable: a triggered check is not a dead end but the start of a structured hunt, and the causes cluster into recognizable families.

Without reproducing their taxonomy in full, the recurring culprits are the ones any small-business tester should recognize:

• Assignment and redirect bias: one variant is served through an extra redirect or a slower path, so a fraction of its users drop out before they are ever counted, thinning that arm.
• Instrumentation and logging gaps: the event that records a user fires reliably in one variant but not the other, so the counts diverge for measurement reasons rather than behavioral ones.
• Bot and non-human traffic filtering that is applied unevenly across variants, removing users from one arm more aggressively than the other.
• Interference from carryover or shared state, where users are not cleanly and independently assigned, so the groups leak into one another.
• Starting the count at the wrong moment, for example triggering analysis on an event that itself is affected by the change under test, which biases who enters the experiment.

A/B test validity is a property of the setup, not the outcome

The deeper lesson SRM teaches is about where validity lives. It is tempting to treat A/B test validity as something you assess at the end, by looking at the size of the effect and the confidence interval. SRM proves that some of the most important validity conditions are fixed the moment the experiment is configured and cannot be repaired by analysis.

A clean random split, correct and symmetric instrumentation, and an assignment unit that does not leak are not outputs of a test. They are preconditions. When they hold, the statistics you compute mean what they claim. When they do not, the same statistics are precise measurements of the wrong thing. This is why experienced experimenters read the health checks first: a beautiful, significant lift sitting on top of a failed SRM check is not a weak result to caveat, it is a void one to discard.

Trustworthy experiments were defined by the people who ran them at scale

It is worth being clear that none of this is a fringe statistical objection. The framework of trustworthy experiments, the idea that an A/B result earns belief only after passing a defined set of validity checks, comes from the practitioners who built and ran experimentation platforms at the largest scale, and who codified it precisely because they watched confident results collapse under scrutiny.

One of their governing heuristics is Twyman's law: any figure that looks surprising or unusually good is probably wrong, and should be checked before it is trusted. A dramatic lift is a reason for more suspicion, not less. SRM is one of the concrete checks that Twyman's law motivates, sitting alongside guarding against peeking, novelty and primacy effects, and network interference. The common thread is a discipline that assumes a result is broken until it has survived the checks that could break it.

An A/B testing checklist begins before launch

If SRM can invalidate a test before it starts, the practical response is to move the decisive work to before launch. A trustworthy A/B testing checklist front-loads the choices that determine validity, so that when the data arrives there is nothing left to negotiate.

• Write the hypothesis and name one primary metric before the test goes live, so success cannot be redefined afterward by picking the chart that happens to look best.
• Calculate the required sample size and run length in advance from the baseline rate and the smallest effect worth acting on, rather than stopping whenever the numbers look good.
• Verify that assignment is random and that tracking fires once, on the right action, symmetrically across every variant.
• Run the SRM check as a gate before reading the outcome, and treat a failure as a hard stop that sends you to diagnose the cause, not a caveat to note.
• Fix the stopping rule in writing to prevent peeking, since continuously monitoring a fixed-horizon test and stopping early inflates the false-positive rate well beyond the nominal threshold.

What SRM actually shows

Sample Ratio Mismatch does not say that A/B testing is unreliable. It says the opposite: that experimentation is trustworthy exactly to the extent that it is disciplined, and that the discipline is checkable. The check is cheap, it is well defined, and it exists precisely so that the good tests can be believed and the compromised ones can be caught early rather than shipped.

For a small business, the takeaway is not to run fewer tests. It is that a test is only worth acting on if the setup around it holds, and that the setup is where the work quietly is. The number you act on is only as good as the split that produced it.