Incrementality Testing With Geo Experiments: Proving a Campaign Worked Without a User-Level Pixel

The measurement you actually want is a counterfactual

Almost every number a business sees about its advertising describes a correlation: this many people who saw the ad went on to buy. The number a business should care about is a counterfactual: how many of those buyers would not have purchased if the ad had never run. The first is what happened alongside the campaign. The second is what the campaign caused. They are not the same quantity, and the difference is not academic.

The reason the two diverge is selection. Ad systems are built to show impressions to the people most likely to convert, and retargeting and branded search in particular reach audiences who were already on their way to buying. An attribution model that credits those conversions to the ad is measuring the platform's targeting skill, not its persuasive effect. Isolating the causal share requires a comparison group that did not receive the campaign but is otherwise like the group that did.

This is the same logic that underpins a randomized controlled trial in medicine. The treatment effect is defined as the difference between a treated population and an equivalent untreated one. The whole discipline of incrementality testing is the effort to construct that untreated comparison honestly when the unit being treated is a market rather than a patient.

Why observational attribution keeps getting it wrong

The strongest evidence that correlation-based measurement misleads comes from inside one of the platforms that sells it. Gordon, Zettelmeyer, Bhargava and Chapsky ran fifteen large-scale randomized field experiments at Facebook and compared the true causal lift each experiment produced against what standard observational methods, the same family that underpins most multi-touch attribution and lookalike modeling, would have estimated from the identical data.

The observational estimates frequently landed in the wrong direction or the wrong magnitude, and conditioning on rich demographic and behavioral covariates did not reliably fix the gap. This matters because it means the failure is not a data-quality problem that better tracking would solve. It is structural. The methods answer the wrong question, so feeding them more individual-level signal does not make the answer right, it makes a confident wrong answer.

That finding reframes the privacy conversation entirely. If the pixel-based estimate was already systematically biased, then losing the pixel is less a catastrophe than an occasion to adopt the method that was more trustworthy all along.

The pixel was fraying before privacy finished it

Two forces are pulling user-level tracking apart at once. The first is regulatory and platform-driven. Apple's App Tracking Transparency, introduced with iOS 14.5 in April 2021, required apps to ask permission before tracking users across other companies' apps and sites, and industry measurement reports put the opt-out rate high enough that a large share of the mobile signal simply stopped flowing. Pixel-based revenue attribution for e-commerce, by the same vendor reporting, fell from capturing most conversions to capturing a clear minority.

The second force is the deprecation and partial replacement of third-party cookies, which removed much of the cross-site scaffolding that desktop attribution relied on. The industry response has been to move toward server-side and consented signals and, more importantly for this discussion, toward aggregate methods that never depended on identifying individuals in the first place: marketing mix modeling and geo experiments.

The opt-out and attribution-loss percentages that circulate come from ad-tech vendors with a commercial interest in the narrative, not from independent peer review, so they are best read as directional rather than precise. What is not in dispute is the direction and the dates: individual-level measurement is materially weaker than it was, and it is not coming back.

How a geo experiment works

A geo experiment abandons the individual as the unit of measurement and adopts the geographic market instead. The advertiser divides its footprint into regions, runs the campaign in some of them, deliberately withholds it in others, and measures the difference in outcomes between the two sets. Because the holdout regions never saw the campaign, they stand in for the counterfactual: what the treated regions would have done anyway.

What the evidence says about geo methods

The method itself is established: synthetic-control geo experimentation is used across the advertising industry and is the design that both platform measurement teams and independent practitioners reach for when a pixel is unavailable or untrusted. The comparative performance numbers, however, deserve a more careful label.

An independent head-to-head simulation study by Recast compared several open-source geo-testing tools and reported that GeoLift's coverage, the share of simulations in which its interval contained the true effect, came closest to the 95 percent target at roughly 92 to 95 percent, and that its false-positive rate, the share of simulations in which it detected an effect that was not there, was the lowest of the tools compared at roughly 3 to 5 percent. Those are encouraging results, and they are the kind of validation the field needs.

They are also a single vendor-run simulation rather than a peer-reviewed benchmark. The right reading is that the numbers support the method's reliability in the tested conditions without settling it as a law. A geo test is a strong instrument; it is not an oracle, and its accuracy on any given account depends on how well the pre-period matched, how large the true effect is relative to market noise, and whether the test had the statistical power to see it.

Geo experiments and marketing mix modeling are complements, not rivals

Geo experiments belong to a broader family of aggregate, privacy-durable measurement methods that survived the collapse of user-level tracking precisely because they never used it. Marketing mix modeling is the other member of that family. Rather than following individuals, it regresses an outcome time series such as sales on marketing time series, with adstock and saturation transforms to model carryover and diminishing returns, and recovers each channel's contribution statistically.

The clearest signal of how seriously the industry now takes these methods is that both of the largest ad platforms have open-sourced their internal versions: Google's Meridian, built on a geo-level Bayesian hierarchical media-mix model, and Meta's Robyn. That the companies whose businesses depend on advertising being seen to work have published the aggregate methods that grade advertising more skeptically than a pixel does is itself evidence of where credible measurement is heading.

The two methods answer different questions and are strongest together. A geo experiment gives a clean causal read on one campaign or channel over one window. A mix model gives a continuous, whole-account picture across channels and time, calibrated by the experiments. Neither replaces the other, and a serious measurement practice runs experiments to anchor the model rather than choosing between them.

The traps that make a test lie

A geo experiment is only as honest as its discipline. The catalog of ways controlled experiments mislead is well documented by Kohavi, Tang and Xu, drawing on the experience of running more than twenty thousand experiments a year across Google, LinkedIn and Microsoft. The most common is peeking: checking results before the test reaches its planned sample and stopping early the moment they look significant, which manufactures false positives because random noise will eventually cross a threshold if you keep looking. Sample-ratio mismatch, novelty and primacy effects, and misread interactions round out the list.

For a single local business there is a harder, quieter constraint underneath all of these: statistical power. Standard sample-size math at conventional significance and power levels, for a minimum detectable effect in the single-digit percentages, calls for tens to hundreds of thousands of observations. At the traffic a single-location business realistically sees, detecting even a sizeable relative lift can require many months, well past the point where the campaign or the business has already changed. This is a structural reason, not a moral one, that owner-operated firms have so often been sold vanity metrics instead of causal ones.

The resolution is not a shortcut. It is choosing tests scaled to the question, accepting that some effects are too small to prove at a single site in a useful window, and, where possible, pooling evidence across many similar businesses to recover the statistical power that no single one of them has. An experiment you were not powered to read is not evidence; it is a coin flip wearing a lab coat.

What a geo experiment cannot tell you

The method has real limits. A geo test measures the aggregate effect of what ran in the treated markets during the window; it does not attribute that effect to a specific creative, keyword or audience unless the design isolates one. It assumes the holdout markets were not contaminated by spillover, which cross-border media and word of mouth can violate. It measures the period it ran, not next quarter, and a lift that holds in spring may not hold in a promotion-heavy December.

It also sits beside a genuinely unsolved measurement frontier. Whether and why an AI answer engine cites a given business is a second attribution problem being invented from scratch, and there is no agreed sampling methodology for it yet, in part because generative systems are non-deterministic by construction. Geo experiments are the mature end of the measurement world; share-of-answer measurement is the immature end. A serious practice states which is which rather than lending the maturity of one to the other.

None of this is a reason to prefer the confident wrong number. It is the reason to prefer the bounded one, and to keep re-running it as accounts and platforms change.