Vanity Metrics, Decision Metrics: A Field Guide for Owners Who Do Not Have a Data Team

The metric you were sold is rarely the metric you can act on

The oldest complaint in marketing is that you cannot tell which half of your spending works. The line is usually put in John Wanamaker's mouth, though it has never been traced to him: the earliest documented match is a secondhand 1919 speech, and the same sentiment has been credited to William Lever and William Wrigley too. The industry's founding parable about measurement is itself unverified, which is a fitting place to start a discussion about the numbers you are handed.

The pattern the parable names has repeated for a century. A convenient, computable number gets adopted at scale, gets treated as truth for years, and only gets corrected once someone runs the harder experiment or an industry votes to discipline itself. The public-relations trade did exactly that when it renounced Advertising Value Equivalency, a metric that priced coverage as if it were bought advertising. A vanity metric, in short, is any number that is easy to produce, pleasant to report, and disconnected from a decision.

A decision metric passes a harder test. If it moves, you would do something different, and you can name what. It resists gaming, it points at an outcome rather than an output, and it survives being read by someone who did not want the answer. Everything below is built to that standard.

Why owner-operators get sold vanity metrics: a scale problem, not a moral one

There is a structural reason a single-location business gets handed impressions instead of proof, and it is not that agencies are lazy. Rigorous measurement has a scale floor most local businesses fall beneath.

Consider the tool everyone reaches for first, the A/B test. Standard sample-size math at 95 percent significance and 80 percent power, chasing a realistic 5 to 10 percent minimum detectable effect, typically calls for tens to hundreds of thousands of visitors. At the traffic a real med-spa or home-services site sees, roughly a thousand visitors a month, detecting even a 20 percent relative lift can take more than seven months, and a 10 percent lift more than two and a half years, well past the point where the season, the offer, or the business itself has already changed.

The same floor sits under the famous brand-building findings. The 60:40 split and the double-jeopardy law were derived from large, multi-brand, multi-decade datasets, not from single small firms. This is why owner-operators end up with vanity metrics: the firm-level experiment they would need is structurally out of reach, so they are sold the number that is easy to compute instead. The real resolution is not a shortcut. It is aggregation, pooling many similar businesses to recover statistical power no single one of them has, and choosing metrics that do not require a randomized trial to read.

The test of a decision metric: could it change what you do on Monday?

Before naming the four, it helps to state the discipline, because the same discipline is what a serious measurement standard already demands. When the communications industry rewrote its rules as the Barcelona Principles, the core instruction was to measure outcomes, not outputs, and to treat measurement as something that must be transparent, consistent, and valid rather than a flattering number.

Translated for an owner, a decision metric has four properties. It is actionable: a move in it has an obvious next step. It is honest: it is never reported naked, always with its method and its comparison. It is hard to inflate: you cannot juice it without doing the underlying work. And it is tied to a buyer, not to your own activity. Impressions fail every test. The four metrics below pass them.

One decision metric per Machine-Readiness Score pillar

The Machine-Readiness Score reads your visibility across four pillars, because those are the four places a buyer now decides: classic search, the local map pack, AI answers, and reputation. Each pillar has exactly one number worth acting on. Track these four and you have a truer picture than a forty-line dashboard gives.

The double-jeopardy trap in every reputation number

One statistical law deserves its own section because it quietly distorts how owners read their reputation pillar. Double jeopardy is one of the most replicated regularities in marketing science: brands with smaller market share have both fewer buyers and slightly lower loyalty among the buyers they do have. It has held across packaged goods, banking, insurance, and newer categories like streaming and ride-share.

Applied to your review counts, the lesson is not to panic when the larger competitor has more reviews and a marginally higher average. That is the expected pattern for a smaller player, not evidence you are worse. The decision this metric should drive is a steady program to raise real review velocity, not a despairing comparison of lifetime totals. Reading double jeopardy correctly is the difference between acting and flinching.

The traps even good metrics fall into

A decision metric can still be misread. The most common failure is peeking: checking a test or a trend the moment it looks like it is working and acting on it early. In large-scale industrial experimentation this is a catalogued way to convince yourself of an effect that is not there, alongside sample-ratio mismatch and novelty effects. For an owner the practical rule is to let a change run a fair, pre-decided window before you judge it, and to compare against a baseline rather than against your hopes.

The second trap is the naked number. A metric reported without its method and its comparison is not a decision metric, it is decoration. "Views up 30 percent" means nothing without the window, the source, and what it is up against. Every number in this guide is meant to be carried with its method attached, which is also the rule the Machine-Readiness Score itself is built on.

What to stop tracking

The corollary of four decision metrics is a longer list of numbers you can safely ignore. None of these should drive a decision on their own.

• Total impressions or total ad impressions, which rise with irrelevant reach.
• Raw website sessions, which mix bots, brand searches, and idle curiosity with buyers.
• Follower and like counts, which almost never move booked business.
• An advertising-value-equivalent for press coverage, a metric its own industry formally abandoned.
• Lifetime star averages that have not moved in a year, which hide recent trajectory.
• Any single-session A/B test result read early, which the statistics say you cannot yet trust.

The limit of this guide

Three of the four decision metrics rest on settled ground: the scale floor under small-business testing, the double-jeopardy law, the abandonment of vanity metrics like the advertising-value-equivalent, and the endorsement rules are all established. The fourth, share of answer, is genuinely emerging, and the correct posture is transparency about that immaturity rather than a claim to have solved it. The Machine-Readiness Score follows that same corrective tradition: a small set of measured numbers, each reported with its method, which is the most an owner without a data team should ever act on.