Does Marketing Science Replicate? A Close Look at the Evidence Behind "What Works"

The replication gap marketing rarely mentions about itself

Replication, rerunning a study to see whether its result holds, is how a science tells durable knowledge from a lucky first result. It is unglamorous and it is the load-bearing wall of the whole enterprise. A finding that no one has ever reproduced is not yet knowledge; it is a hypothesis that got published.

By that standard marketing has a problem its own scholars named long ago. In a 2007 analysis in the Journal of Business Research, Evanschitzky, Baumgarth, Hubbard and Armstrong revisited the state of replication in the discipline and documented what they called a disturbing trend: marketing publishes markedly fewer replication studies than the natural sciences, and the share was not improving over time. The consequence is structural. If a field rewards novel findings and rarely funds or publishes the boring work of retesting them, most of its accepted results are, quite literally, untested past their first appearance.

This is not an accusation of fraud. It is a description of an incentive system. The point for a business owner is narrower and more useful: when a marketer cites a study to justify a tactic, the relevant question is "has anyone ever reproduced it," not just "is the study real," and for a large fraction of marketing claims the answer is no.

The replication crisis, and why priming is the cautionary tale

Marketing does not live alone. A great deal of its persuasion advice, especially the neuromarketing and behavioral-science genre, is imported from academic social psychology, and that field spent the last decade in a public reckoning now known as the replication crisis. Large coordinated efforts to rerun well-known psychology experiments found that a substantial portion did not reproduce, or reproduced with effects far smaller than the headline studies claimed.

The instructive case is priming, the idea that a subtle cue can meaningfully shift later behavior without a person noticing. Priming effects have been invoked constantly in marketing-psychology claims, and several prominent ones suffered high-profile replication failures, with real-world effect sizes proving far less reliable than the original lab findings suggested. That does not mean every priming claim is false. It means the category earned its skepticism, and any confident marketing tactic built on a single striking priming study is standing on ground the psychologists themselves stopped trusting.

The lesson transfers cleanly. When a pitch leans on "studies show that people subconsciously..." the correct reflex is to ask which studies, how large, and whether they held up when someone tried again. Often the trail ends at one memorable experiment that never survived a serious retest.

What does survive replication: the laws that keep holding

The record does not stop at the failures. Some marketing findings are among the most replicated regularities in all of social science, and they deserve to be trusted precisely because they were retested across categories and decades.

What happens when a plausible metric meets a real experiment

The clearest way to see the difference between folklore and finding is to take a measurement everyone trusts and test it against an experiment. Gordon, Zettelmeyer, Bhargava and Chapsky did exactly that with advertising attribution. Using 15 large-scale randomized field experiments at Facebook, drawn from more than 500 million user-experiment observations and 1.6 billion ad impressions, they compared the true causal lift an ad produced against what standard observational attribution methods, the machinery behind most multi-touch attribution, would have estimated from the same data.

The observational methods frequently got it wrong, sometimes in the wrong direction or the wrong magnitude, even after controlling for rich demographic and behavioral data. In other words, a widely sold, computationally convenient way of measuring "what works" did not survive contact with a randomized experiment. This is the same failure mode as the replication gap, just at the level of a metric rather than a paper: a plausible number gets adopted at scale, gets treated as truth, and only gets corrected when someone runs the harder test.

The industry has a habit of retiring its own bad metrics

There is a hopeful side to this story, and it is the tradition of self-correction. Marketing and its adjacent industries do, periodically, admit a favorite metric was never valid and vote to retire it.

The public-relations industry did this openly. For years the standard way to value coverage was Advertising Value Equivalency, a made-up number that priced editorial coverage as though it were paid advertising. In 2010, and again in the revised Barcelona Principles of 2015 and 2020, the industry's own measurement body formally rejected AVE in favor of outcome-based measurement, calling instead for metrics that are transparent, consistent and valid. An entire field looked at its flagship number and called it a vanity metric to its face.

Even the industry's founding parable about measurement turns out to be unverified. The famous line, "half my advertising is wasted, I just don't know which half," is popularly pinned on the retailer John Wanamaker, yet the earliest documented match is a secondhand 1919 account and the same sentiment has been attributed to several other men. The one story marketing tells about the honesty of measurement cannot itself be sourced. That is not a reason for cynicism. It is a reason to treat confident attribution, of quotes and of results, as a claim to be checked rather than a fact to be repeated.

How to read a marketing claim like a scientist

None of this requires a statistics degree. It requires a short, repeatable set of questions that separate a durable finding from a confident guess.

• Ask whether it has ever been replicated. A single striking study is a starting point, not a conclusion. The findings you can lean on, double jeopardy, buyer-acquisition patterns, were retested across many categories and decades.
• Weigh the source data against the claim. A pattern derived from hundreds of large national brands may simply not transfer to a single-location business, and an honest presenter says so rather than smoothing it over.
• Distrust the round, memorable number. When a figure like "accounts for 83% of the variance" comes from one industry analysis of a few dozen cases, it is a single result, not a settled law, and should be labeled that way.
• Prefer experiments to observation. A randomized or geo-based test that withholds treatment from a control tells you what an intervention caused; a dashboard that correlates spend with outcomes usually cannot.
• Watch for the vanity metric. If a number is easy to compute and always goes up, be suspicious. The metrics worth acting on are the ones that can also deliver bad news.

What this means for buying marketing services

The takeaway is not nihilism about marketing. It is a posture. Some of what a good marketer knows is genuine, replicated science; some of it is untested convention repeated with a straight face; and the two are usually delivered in the same confident tone. A firm worth hiring is one that can tell you, without prompting, which of its claims rests on replicated evidence, which rests on a single study, and which rests on nothing firmer than industry habit.

The Machine-Readiness Score, a four-pillar read of where a business stands across classic search, the local map pack, AI answers and reputation, sits inside this corrective tradition rather than outside it: a published method, sourced figures, and clear labels on what is measured versus what is still an emerging estimate.