Why Your Google Ads Dashboard Overstates Its Own Impact: Lessons From a Randomized Field Experiment at eBay

The dashboard answers the wrong question

A Google Ads report is a record of clicks that were followed by conversions. When a buyer clicks an ad and later purchases, the conversion is assigned to that click, and the account's return on ad spend is computed from the assigned conversions. The arithmetic is correct. The inference most owners draw from it is not.

The number every account is judged on answers the question, how many sales came after an ad click? The question a budget owner actually needs answered is different: how many sales happened because of the ad that would not have happened otherwise? The first is a matter of counting. The second is a matter of causation, and causation cannot be read off an observational report no matter how granular the report is.

This is the specific failure the eBay experiment was built to expose. It is also why measurement scientists insist that a controlled experiment, not a richer attribution model, is the only instrument that recovers the causal number.

What Blake, Nosko and Tadelis actually did at eBay

In "Consumer Heterogeneity and Paid Search Effectiveness: A Large-Scale Field Experiment," published in Econometrica in 2015, Thomas Blake, Chris Nosko and Steven Tadelis used eBay's own scale to run the test that observational data cannot substitute for. Rather than infer the effect of paid search from historical clicks and conversions, they randomized. For selected keywords and geographies, paid-search ads were switched off for randomly assigned groups of users while remaining on for others, creating a genuine treatment and control.

With that design in place, the counterfactual is observed rather than assumed. The purchases that still occurred in the groups with ads turned off are, by construction, the purchases that did not need the ad. The incremental effect of paid search is the difference between the groups, and nothing about a user's prior intent can contaminate it, because assignment to treatment or control was random.

The authors then compared this experimental estimate against the kind of non-experimental estimate a standard attribution report produces. The observational method credited paid search with far more value than the randomized experiment could find. The divergence was not noise. It was the structural bias that any last-click or click-attributed report carries.

Brand keyword bidding: paying to be found by people already looking for you

The sharpest result concerned brand keywords, the search terms that contain the advertiser's own name. Intuitively these look like cheap, high-converting inventory, and in a dashboard they are: someone searching for "eBay" who clicks an eBay ad and buys will be recorded as a conversion driven by the ad.

The experiment showed this to be an accounting illusion for a well-known brand. Brand-keyword ads produced no measurable incremental short-term benefit once tested against a randomized control. The people clicking those ads were, in the main, people already navigating to the site. The ad intercepted demand that the organic listing directly beneath it would have captured for free. The conversions were real; the incremental conversions caused by the ad were, within the study, not distinguishable from zero.

This finding needs a boundary drawn around it, addressed in full below. eBay is one of the most recognized commerce brands in the world, and the null result is strongest precisely where a brand is already the destination a searcher has in mind. It does not license a blanket claim that every business should abandon brand-keyword ads. It does mean that the value a brand-term campaign reports is the most likely of all paid tactics to be borrowed from demand you already owned.

Why last click attribution bias inflates the number

The mechanism is selection, not fraud. Advertising platforms are extremely good at showing ads to people who are likely to convert. That is the product working as designed. But it means the population that sees and clicks an ad is not a random sample of the market; it is enriched with people who were already close to buying.

When a report attributes their conversions to the click, it conflates two effects that only an experiment can separate. One is persuasion: the ad genuinely moved someone who would not otherwise have bought. The other is selection: the ad was simply present at the moment a committed buyer was going to act anyway. Observational attribution sums the two and prints the total as if it were all persuasion. The eBay experiment removed the selection component by randomization, and a large part of the reported return went with it.

This is why adding more touchpoints does not fix the problem. Multi touch attribution distributes credit across several clicks rather than only the last, which changes how the inflated total is divided but not the fact that the total itself is inflated by selection. A model that only ever sees users who were shown ads cannot, by construction, observe what those users would have done with no ads at all. Only a holdout can.

The gold standard is an experiment, not a better attribution model

The eBay study sits inside a broader consensus in the measurement-science literature: the credible way to know whether an intervention caused an outcome is to run a randomized controlled experiment with a holdout, then check the experiment itself for the errors that quietly invalidate results.

Ron Kohavi, Diane Tang and Ya Xu, drawing on more than twenty thousand controlled experiments run annually at Microsoft, codify this discipline in their 2020 practitioner text. Two of their warnings are directly relevant to anyone tempted to trust a flattering ad report.

Reading incremental ROAS and MER

If the platform number overstates impact, what should a budget owner look at instead? Two more reliable instruments, neither of which a single-campaign dashboard shows by default.

The first is incremental ROAS, sometimes written iROAS. Where ordinary return on ad spend divides all attributed revenue by cost, incremental ROAS divides only the revenue the ads actually caused, established by a holdout, by cost. It is almost always a lower number than the platform figure, and it is the more accurate one. The eBay result is, in effect, a demonstration that for some campaigns, brand terms in particular, incremental ROAS can be dramatically below the reported ROAS.

The second is the media efficiency ratio, or MER, which is total revenue over total advertising spend across the whole account rather than per campaign. MER is deliberately blunt. It cannot be gamed by one campaign claiming credit for another's demand, because it never tries to attribute at the campaign level. Watched over time against spend changes, it exposes the situation the eBay study warns about: spend rises, the per-campaign dashboards stay green, and blended revenue barely moves, which is the signature of paying for demand you already had.

Neither instrument is a silver bullet, and reliable measurement usually combines a blended top-line view with periodic holdout tests on the campaigns large enough to justify them. What both instruments share is that they refuse to treat a click that preceded a sale as proof that the click caused the sale.

What this study proves, and what it does not

The eBay experiment is among the strongest causal-inference results available in marketing measurement: a real randomized field experiment at large scale, published in a top-tier peer-reviewed economics journal. Its core finding, that observational attribution inflates paid-search ROI because clicks select for pre-existing intent, is established and has been influential precisely because the method leaves little room to argue with the direction of the bias.

Its limits deserve the same scrutiny. The experiment measured one advertiser, eBay, a brand with enormous unaided awareness, and the starkest null result is for the case where that awareness matters most, brand keywords. A smaller or less-known business bidding on its own name may be defending against competitors' ads or reaching people who genuinely would not have found it otherwise, and the incremental value there is an empirical question for that business, not a foregone conclusion. The study also measured short-term effects; long-run brand consequences of turning ads off are outside its window.

The correct reading is therefore narrow and durable at once. Narrow, because the exact magnitude does not transfer to your account. Durable, because the mechanism does: any report that counts conversions after clicks, without a holdout, will overstate the causal contribution of advertising to people who were already going to convert. The remedy is not to distrust all advertising. It is to measure it the way the evidence says it must be measured.