Geo-Experiments: The Randomized Controlled Trial for Local Advertising

The number your ad platform reports is not the number you want

An advertising dashboard reports a correlation. It records that a set of people saw an ad and that some of them later converted, and it credits the ad with those conversions. What it cannot see is the counterfactual: how many of those same people would have converted anyway, with no ad at all. The reported figure and the causal figure are different quantities, and for most campaigns the reported one is larger.

The size of that gap is not a matter of opinion. In a set of three controlled experiments, Randall Lewis, Justin Rao, and David Reiley documented what they called activity bias: the surprising degree to which a person's online behaviors are correlated in time. People who happen to be browsing are also more likely to be searching, clicking, and buying in the same window, advertisement or not. Any measurement that compares exposed users to a naively chosen unexposed group inherits that correlation and reads it as ad effect. The result is a systematic overestimate of what advertising did.

This is the problem a geo-experiment is built to solve. Rather than trusting the platform's attribution, it manufactures a genuine control group and reads the difference directly.

What a geo-experiment actually is

The design borrows its logic directly from the clinical trial. In medicine, you cannot know what a drug did to a single patient, because you never observe that patient both treated and untreated. You solve the problem at the level of groups: randomly assign patients to a treatment arm and a control arm, and the random assignment makes the two groups statistically comparable in everything except the treatment. The difference in outcomes is then attributable to the treatment.

Why geography measures lift without tracking a single person

The quiet advantage of the geo design is that it never needs to identify anyone. It reads outcomes at the level of the region, total conversions in region A versus total conversions in region B, so it does not depend on cookies, device graphs, or individual-level identity resolution at all. The randomization does the work that individual tracking is usually asked, and often fails, to do.

That property has become more valuable, not less, as the tracking ground has shifted underneath advertisers. Google spent six years building its Privacy Sandbox as a replacement for the third-party cookie, then terminated the initiative in 2025, retiring the remaining APIs and leaving third-party cookies in Chrome with no removal timeline. The lesson for a measurement strategy is that platform-level tracking policy is not a stable foundation to build on. A method whose validity comes from randomizing regions rather than from following users is insulated from that volatility by construction.

The evidence that reported lift and real lift diverge

The reason to go to this trouble is that holdout experiments repeatedly find less lift than dashboards claim, and sometimes find none. The clearest demonstration in the paid-search literature is a large-scale randomized field experiment run at eBay by Thomas Blake, Chris Nosko, and Steven Tadelis. Advertising on the company's own branded keywords produced no measurable short-term incremental benefit: the users clicking those ads were people already headed to eBay, who would have arrived through organic results or direct navigation regardless.

For non-brand keywords the picture was more mixed and more instructive. New and infrequent users were positively influenced by the ads, but frequent users, whose purchases the ads did not change, absorbed most of the spend, dragging the average return negative. A single blended dashboard number would have hidden that entirely. Only a design that separated the causal effect from the correlational one could surface it, which is exactly what a geo-holdout does at the regional level.

This is a single-firm study, and its specific magnitudes should not be transplanted onto a med-spa or a plumbing company as if they were constants. What generalizes is the mechanism: the more an ad is shown to people who already intended to buy, the more attribution overstates it, and branded and retargeting inventory is where that overlap is highest.

Cheaper cousins: ghost ads and the cost of causal measurement

Geo-experiments are not the only causal instrument, and they are not always the cheapest. The older gold standard, the public-service-announcement holdout, showed the control group an unrelated charity ad in place of the real one so the two groups were treated identically except for the message. It works, but it is expensive, because the advertiser pays to serve placebo impressions.

Garrett Johnson, Randall Lewis, and Elmar Nubbemeyer introduced a lower-cost alternative they called ghost ads: rather than buying placebo placements, the system records the counterfactual impressions a control user would have been served, and compares against those. On a retargeting campaign, their method measured a lift of 17.2 percent in site visits and 10.5 percent in purchases while working natively with real-time ad-delivery optimization and at a fraction of the holdout cost.

The point for a local advertiser is not that one method wins. It is that a whole toolkit exists for answering the causal question, and that geo-experiments occupy the position in that toolkit where the business runs across separable regions and wants an answer that owes nothing to individual tracking.

What a geo-experiment can and cannot tell you

The design has real preconditions, and acknowledging them is part of using it well. It needs enough non-overlapping regions to randomize, which means a genuinely single-location business cannot run a clean version of it. It needs the regions to be reasonably separable, so that turning ads off in one place does not simply push demand into another. It needs a test window long enough to accumulate signal, because the incremental effect is usually a modest fraction of total sales and modest effects require patience and volume to detect.

It is also worth being precise about what the wider industry claims here versus what is established. Practitioner analyses frequently report that measured incremental ROAS runs 30 to 70 percent below platform-reported ROAS, with branded search the worst offender. That direction is consistent with the peer-reviewed findings above, but the specific percentage ranges come from unaudited vendor case studies, not academic replication, and should be treated as directional rather than as a number to quote as fact. The correct posture is to measure your own gap, not to import someone else's.

Reading the result

A geo-experiment returns an incremental figure that will almost always be lower than the platform's attributed figure. That is the finding, not a failure of the test. The disciplined response is to treat the incremental number as the real one and to re-plan bids and budgets against it, which is precisely the decision use Vaver and Koehler had in mind.

It also explains the growing practitioner preference for a blunt anchor metric. Marketing Efficiency Ratio, total revenue divided by total marketing spend, cannot be gamed by shifting attribution credit between channels, because it never assigns channel-level credit in the first place. It is an industry construct rather than an academic one, but its rationale is the same activity-bias and attribution-gaming logic the field-experiment literature documents. A geo-experiment tells you the causal contribution of a specific lever; a blended ratio keeps you honest about the whole. Neither is a naked platform-reported ROAS, and that is the point.

Who should run one

The design fits a particular shape of business. If you operate across several markets, a multi-location med-spa group, a home-services company covering many metros, a franchise system, you already have the raw material a geo-experiment needs, and you are also the kind of advertiser for whom a persistent attribution gap compounds into real money over a year of spend.

You do not need a statistics department to benefit from the method. You need someone to design the test so the regions are comparable, hold the discipline of the holdout, and read the result without flattering it. That measurement practice is what decides what to keep spending on and what to cut.