Metrics That Matter: Vanity vs Actionable Metrics

Some metrics make a business feel good and tell it nothing. Total sign-ups, cumulative downloads, page views and follower counts tend to rise, look impressive in presentations, and rarely change a decision. The lean startup movement named these vanity metrics: measurements that give "the rosiest picture possible" but do not reflect the key drivers of a business. Actionable metrics, by contrast, lead to informed decisions and action (Wikipedia: lean startup). Eric Ries, who popularised the idea, argued that companies should look past gross outputs to the drivers that show whether they are actually making progress (Ries: beware of vanity metrics).

The difference is easiest to see with real data.

A total that hides a problem

A UK online gift retailer's transactions for 2011 (UCI: Online Retail) can be summarised in two ways: the number of customers the business has ever had, or what happened to customers in each month.

import io
import urllib.request
import zipfile
import pandas as pd

url = "https://archive.ics.uci.edu/static/public/352/online+retail.zip"
with zipfile.ZipFile(io.BytesIO(urllib.request.urlopen(url).read())) as archive:
    raw = pd.read_excel(archive.open("Online Retail.xlsx"), dtype={"InvoiceNo": str})
sales = raw[~raw["InvoiceNo"].str.startswith("C") & (raw["Quantity"] > 0) & (raw["UnitPrice"] > 0)
            & raw["CustomerID"].notna() & (raw["InvoiceDate"] < "2011-12-01")]
sales = sales.assign(month=sales["InvoiceDate"].dt.to_period("M"))
first_month = sales.groupby("CustomerID")["month"].min()
table = pd.DataFrame({
    "customers to date": first_month.value_counts().sort_index().cumsum(),
    "new this month": first_month.value_counts().sort_index(),
    "active this month": sales.groupby("month")["CustomerID"].nunique(),
})
table["returning share"] = 1 - table["new this month"] / table["active this month"]
print(table.loc["2011-01":].round(2).to_string())

         customers to date  new this month  active this month  returning share
month                                                                         
2011-01               1302             417                741             0.44
2011-02               1682             380                758             0.50
2011-03               2134             452                974             0.54
2011-04               2434             300                856             0.65
2011-05               2718             284               1056             0.73
2011-06               2960             242                991             0.76
2011-07               3148             188                949             0.80
2011-08               3317             169                935             0.82
2011-09               3616             299               1266             0.76
2011-10               3974             358               1364             0.74
2011-11               4297             323               1664             0.81

"Customers to date" rises every single month, from 1,302 to 4,297. On a slide it would look like steady success. Yet new customers per month fell from 452 in March to 169 in August, a drop of 63%, before the autumn season revived them. A cumulative total can only go up, so it cannot show a slowdown; it would have hidden a five-month decline in acquisition from anyone who watched only that line. December 2010 is left out because the data begins that month, so its "new" customers include everyone who happened to buy then.

The monthly figures are actionable. Falling new customers point at acquisition: channels, campaigns, search visibility. A rising returning share, from 44% to over 80%, shows the existing base carrying the business, which is good news for retention and a warning about dependence on it.

What makes a metric actionable

• It changes decisions. If it moved sharply, someone would do something different.
• It is a rate, ratio or cohort measure, not a running total. Conversion rate, new customers this month and retention of a cohort can fall; totals rarely do.
• It can be compared with a previous period, a target or another segment.
• Its cause can be traced, because it maps to a lever someone controls.
• It is hard to game without improving the business.

Vanity depends on the business

The same number can be vanity for one company and vital for another. Page views are a weak signal for a company whose revenue does not depend on them, but for an online magazine that sells advertising they are directly linked to revenue (Wikipedia: lean startup). The question is never "is this metric vanity?" but "does this metric connect to how this business creates value?"

Common vanity metrics and better alternatives

A five-question test for any metric

1. What decision would change if this number moved sharply?
2. Can it go down, and would we notice if it did?
3. What do we compare it with?
4. Who owns the lever that moves it?
5. Could someone improve it without improving the business?

A metric that fails most of these belongs in an appendix, if anywhere. Metrics that pass are candidates for KPIs; how to create effective KPIs covers turning them into a KPI tree with owners and targets, and KPI scorecards for executives covers reporting them.

Turning a vanity metric into actionable ones

Take "total app downloads", a favourite in launch announcements. Three changes make it useful:

1. Make it per period. Downloads this month, which can fall and therefore warn.
2. Follow what happens next. The share of this month's downloads still active after 30 days, a cohort measure of whether the product keeps people.
3. Connect it to cost and value. Cost per active user from each acquisition channel, which shows where the next unit of budget should go.

The original number is still there, split into pieces that each point to a decision: acquisition volume, product retention and marketing efficiency. The same pattern works for followers, registrations, page views or any running total.

Why vanity metrics persist

They are easy to collect, they rise, and they make reports look good. Replacing them takes some courage: an actionable metric will eventually show bad news. That is the point. A metric that can never deliver bad news cannot warn you when something is going wrong.

KPI analysis and business reporting are part of the business analytics concepts module of the Data Analytics course.

Sources

• Lean startup: actionable metrics vs vanity metrics (Wikipedia) — Wikipedia
• Entrepreneurs: Beware of Vanity Metrics — Harvard Business Review, 2010
• Online Retail (UCI Machine Learning Repository) — UCI Machine Learning Repository