What Happens When You Test Multi-Touch Attribution Against Reality: The Facebook Field Experiments

The question attribution quietly avoids

Every attribution report answers a question of correlation: of the customers who converted, which marketing touches did they encounter along the way, and in what order. Last-click credits the final touch. Multi-touch attribution spreads the credit across several. Both are reading the path of people who already converted.

The question a business owner actually cares about is causal: of the money spent, how much bought a sale that would not have happened anyway. Those two questions have the same units and look like the same number on a dashboard, which is exactly why they are so easily confused. An ad shown to someone who was already going to buy gets full credit in an attribution model and zero credit in reality. The gap between the two is not a rounding error. It is the entire measurement problem.

For most of the industry's history that gap was assumed to be small, or was simply never tested, because testing it requires something most advertisers never had: a genuine control group of buyers who were deliberately not shown the ad. The Facebook field experiments are important because they built that control group at scale, fifteen times over, and measured the gap directly.

The experiment that put attribution on trial

Gordon, Zettelmeyer, Bhargava and Chapsky designed a study with a rare property: it could measure the same advertising effect two ways at once, on the same people, at the same time. Their 2019 paper in Marketing Science reports fifteen large-scale randomized field experiments run on Facebook's advertising platform, spanning more than 500 million user-experiment observations and 1.6 billion ad impressions.

In each experiment, users were randomly split into a test group that could see the advertiser's campaign and a control group that could not. The difference in outcomes between those two randomized groups is the ground truth, the real causal lift, because randomization is the one method that guarantees the two groups differ only in whether they saw the ad.

Then the researchers did something clever. From the very same population, they estimated the ad's effect a second way, using the standard observational methods that underpin most attribution and lookalike modeling. They fit those models with the rich individual-level data a platform like Facebook actually has, extensive demographic and behavioral covariates, and compared the observational answer to the experimental ground truth.

What the Facebook field experiments actually found

The observational methods frequently failed to recover the experimental ground truth. Across the fifteen studies, effect estimates from the standard observational approaches diverged from the randomized result, at times pointing in the wrong direction and at times off by a wide margin. Critically, the divergence persisted even after conditioning on the rich demographic and behavioral covariates available in the data.

That last point is the one to sit with. The failure was not a data problem that more variables would fix. It was structural. Observational methods have to reconstruct a counterfactual, what these buyers would have done unseen, from people who are systematically different: the people an ad system chose to show the ad to are, by design, the people most likely to convert. No amount of covariate adjustment fully undoes a selection that the ad platform itself engineered.

The authors' conclusion is measured, and worth quoting in spirit rather than overstating: observational methods can sometimes approximate experimental results, but they do so unreliably, and an advertiser cannot know in advance whether a given observational estimate is one of the good cases or one of the badly wrong ones. Unreliable in an unknown direction is, for decision-making, close to unusable.

Why last-click and multi-touch attribution inherit the same bias

Last-click attribution has an obvious flaw: it hands all the credit to the final touch and none to everything that built the demand. Multi-touch attribution was sold as the sophisticated fix, distributing credit across the whole path to purchase. But the Facebook result cuts underneath that debate, because both models share the same original sin. Both are observational. Both credit touches on the paths of people who converted, and neither constructs the control group that would reveal which of those conversions the ads actually caused.

This is why the two most over-credited tactics in almost every account are retargeting and branded search. Both deliberately reach people who are already close to buying. An observational model sees those people convert shortly after the touch and awards a glowing return. An experiment that holds a matched group back frequently finds much of that return was going to arrive regardless. The dashboard is not lying about the arithmetic it performed. It is answering the wrong question with impressive precision.

The practical implication is uncomfortable but freeing: the choice between last-click and a fancier multi-touch attribution model is a smaller decision than it appears. Refining the credit-splitting rules of an observational model does not make the model causal. Only an experiment does that.

What rigorous experiments demand, and where they break

If experiments are the standard, it is worth being equally clear about how hard they are to run well. The canonical practitioner text, Ron Kohavi, Diane Tang and Ya Xu's Trustworthy Online Controlled Experiments, distilled from experience running more than twenty thousand experiments a year at Google, LinkedIn and Microsoft, catalogues the ways controlled tests quietly produce false answers.

The most common is peeking: checking a test before it reaches its planned sample size and stopping the moment it looks significant, which inflates false positives. Others include sample ratio mismatch, where the split between groups is subtly broken, and novelty and primacy effects, where a change looks powerful only because it is new. A controlled experiment is the right tool, but a carelessly run experiment can mislead as confidently as the attribution model it was meant to check.

There is also a scale problem that matters enormously for smaller businesses. Standard statistical power requirements for a clean individual-level test, at conventional significance and power targets, typically call for tens of thousands to hundreds of thousands of users. At the traffic a single-location business realistically sees, an experiment sensitive enough to trust can take many months to conclude, by which point the campaign, the season or the business has already changed. This is not a moral failing on the owner's part. It is arithmetic, and it is why the answer for small firms is rarely "run your own fifteen experiments."

The methods that survived: mix modeling and geo experiments

The response to all this, across the serious end of the industry, has been a return to methods that measure causally without depending on tracking individual people. Two matter most.

Why the signal got worse right when it mattered

The attribution problem did not stay still after 2019. Apple's App Tracking Transparency, introduced with iOS 14.5 in April 2021, sharply reduced the individual-level signal that observational attribution depends on. Industry and ad-tech vendors have reported figures such as a large majority of iOS users opting out of cross-app tracking and material drops in pixel-based attribution accuracy, though these point estimates come from parties with a commercial stake in the narrative and are best treated as directional rather than precise.

The direction, however, is not in dispute, and it sharpens the lesson from the Facebook experiments. Observational attribution was already unreliable when the tracking was rich. As the tracking thinned, the same models were asked to do the same reconstruction with less to work from. This is the structural reason the industry pivoted toward consented server-side signals and aggregate, experiment-led methods: not fashion, but the accumulation of evidence that the individual-tracking approach was fragile even at its peak.

A recurring pattern worth recognizing

The Facebook study is one episode in an older story: the marketing industry repeatedly adopts a convenient, computable metric, treats it as truth for years, and only corrects once someone runs the harder experiment. The industry's own founding parable about waste, the line that half of all advertising is wasted and no one knows which half, is popularly pinned on John Wanamaker but has no verified original source, a fitting emblem for a field whose measurement folklore is itself unverified.

The public relations industry offers the cleanest example of self-correction. Under the Barcelona Principles, an industry standard first ratified in 2010 and revised as Barcelona Principles 3.0 in 2020, communications professionals formally renounced Advertising Value Equivalency, a flattering vanity metric, in favor of outcome-based measurement that is transparent, consistent and valid. An entire industry voted to kill its own comfortable number.

This is the tradition a rigorous measurement practice belongs inside, not outside. The corrective posture is not to claim a metric that finally solves everything, but to disclose the method, separate the measurement from the money that could bias it, and never report a figure without the baseline behind it. In the United States that standard also has a legal edge: the Federal Trade Commission's Endorsement Guides in 16 CFR Part 255 require that testimonials be genuine and that material connections be disclosed, a reminder that inflated or unsubstantiated claims are not merely bad manners.

What this means for a business that is not Facebook

A single owner-operated business will never run fifteen randomized experiments across 1.6 billion impressions. That is the wrong takeaway. The right one is that the standard is set, and any measurement that ignores it should be read with suspicion. When a report claims a precise return from a channel, the question is simply: against what control? If the answer is "none, we credited the touches on the paths of people who converted," then the number is an attribution estimate, not a measured causal result, and the Facebook experiments tell you it can be wrong in a direction you cannot predict.

The workable path for a smaller business is not to abandon measurement but to change what counts as proof: prefer holdout and geo experiments where volume allows, hold the whole account to a blended efficiency figure that no single platform can inflate, report every result with its baseline and its confidence, and treat any dashboard ROAS as a claim to be checked rather than a fact to be banked. None of that guarantees a particular outcome. It guarantees that the number you steer by is one you can actually trust.