The Dangers of Vanity Metrics
Don't get caught up in your own hype
Good news! I’ve teamed up with my buddy Ray Rike from RevOps Squared and
to bring you benchmarking resources on a regular basis. I’m proud to say the Mostly metrics community is now large enough to be considered, as the pros say, “statistically relevant”.Oh, talk dirty to me.
Take ten minutes to complete the performance metrics survey using the button below, and we’ll revert back with data to steer your SaaS rocket ship to the moon.
Vanity metrics are dangerous.
They might make you feel all warm and fuzzy about your business today…
But focusing on them will KILL you in the long run.
I asked 13 of the smartest CEOs and Investors I know for examples of vanity metrics that you should AVOID at all costs.
But first off - what are vanity metrics?
Vanity metrics are numbers that make you look good, but have no material impact on your ability to make good decisions and improve your business.
They are superficial and fail to drive durable revenue growth.
Per Ben Yoskovitz, Founder and Managing Partner of Highline Beta and writer of the superb Substack
:A good metric has 4 key qualities:
Understandable: A good metric is one that’s easy for everyone to understand and track. That allows it to be part of the company’s common language.
Comparative: A good metric allows us to compare things over periods of time to see trends. This is often what we think of when we talk about cohort analysis. For example: Active Users vs. Active Users/Month. If I tell you I have 10,000 active users, it’s difficult to know if that’s good or bad. If I tell you that last month I had 1,000 active users, that’s a 10x increase, and that looks pretty good!
Ratio / Rate: If you take a comparative number and then turn it into a ratio or rate, it becomes even more valuable. Using my example above, instead of Users/Month, I should track % Monthly Active Users. So last month I had 1,000 active users out of 2,000 that joined my platform, which is 50% of monthly active users.
Behavior changing: We already covered this above, but as a reminder: a good metric is one that you use to make decisions. Imagine looking at a metric and thinking to yourself, “If this goes up, stays the same, or goes down, I don’t know what I’d do differently.” ← stop focusing on that metric.
So with that context, let's look at some examples of dangerous vanity metrics:
1/ Daily and Monthly Active Users
From talking to Thiel Fellow and vertical SaaS founder Luke Sophinos (@lukesophinos), DAU and MAU are pretty useless without supporting information or specific rules that allow you to quantify them.
For example, if someone logs into an app every month and doesn’t use it for a specific purpose, then it’s pretty irrelevant.
They must be performing actions linked to the core product’s value to count.
2/ Net Churn
Per Niv Thanabalan (@itsnivt), the CEO of Integral Insights, watch out for subscription or membership based companies that only cite their Net Churn.
If you can't tell exactly how many gross customers are dropping out, you may be masking a larger problem through your client acquisition pace.
3/ Cumulative Metrics
From talking to everyone's favorite Fabio looking CFO, @OnlyCFO, cumulative metrics provide the prettiest looking charts but are among the worst in deriving business insights.
It's near impossible to get much from a chart that can't go down period over period.
It pains me to say that @asmartbear recently pointed this out for my favorite newsletter platform:
4/ Funding Raised
I asked Ben Yoskovitz (@byosko) the Founding partner of of Highlight Beta:
"Is the amount of capital a founder has raised for their startup a vanity metric?"
His reply:
"100% YES."
Why? The amount you raise doesn’t drive learning.
It might give you a chance to run more experiments and learn from them, but usually it leads to spending on stuff you don’t really need.
In other words, money can buy you time.
But it can't buy you execution.
5/ ESG Scores
From talking to Antonio Reza (@theantonioreza), Finance leader at Google, Environmental and Social Good (ESG) scores are potentially even more misleading than Net Promoter Scores (which we'll hit on later)
ESG scores are hard to understand (is 70 good?), you don't really know how they were derived (was it through a standardized questionnaire?) and they don't immediately translate to revenue, or really any operating metric.
But maybe you'll get a pat on the back?
I'm sure this will upset a few people, though.
6/ GitHub Stars
For Open Source companies, GitHub stars are a widely cited vanity metric.
Shomik Ghosh (@shomikghosh21), Partner at Boldstart, a day one partner for Dev First & SaaS founders, sees this a lot.
Not only can these scores be gamed and bought, but they are like twitter followers or discord users - complete vanity.
Plus, a high level of activity doesn't always translate into a high level of meaningful innovation going on.
(Check out Shomik’s newsletter: Software Snack Bites)
7/ Net Dollar Retention above a certain threshold
Kyle Harrison (@kwharrison13) Partner at Contrary is quick to sniff out when advertising an NDR for a small cohort of customers is misleading.
A lot of companies will game NDR by grouping it by customer size.
They’ll be like, “Among customers that have $100K ACV, our NDR is 170%.”
But what % of revenue do those customers represent? 5%?
Plus, they SHOULD have high NDR if they are enterprise customers. That stat most certainly benefits from survivorship bias.
(Check out Kyle’s newsletter:
)8/ Any metric void of segmentation
But segmentation CAN be useful when applied honestly, and not just for marketing purposes, like mentioned above.
Patrick Campbell (@Patticus), who Bootstrapped and sold Profitwell for over $200M to Paddle, points out how any metric on a big data set that isn't segmented is basically worthless from a decision making perspective.
Metrics are only as useful as you actually being able to apply them to run the business, and if you don't segment, you never actually use them. For example - you can’t run the entire business off just one, generic CAC Payback figure. You need to know how long it takes to recoup your Customer Acquisition Cost across the various go to market segments of your business, which have different inputs (costs) and outputs (deal sizes).
This applies to most high level metrics like MRR/ARR, LTV, CAC, etc.
9 / Customer Lifetime Value (LTV)
From talking to David Kellogg (@Kellblog), the SaaS Metrics original gangster, LTV is becoming a vanity metric.
This is because people cherry-pick the churn rate (net vs. gross, ATR vs. whole ARR pool), and then invert it.
Furthermore, LTV can also be super misleading if it's being applied to too small of a data set.
Customers need an opportunity to actually churn before you count them.
And typically a track record of under three years is rubbish, which means you have to rely on assumptions.
10/ Headcount
HC on its own is just a reflection of cost without any context to productivity.
Anand Sanwal (@asanwal), CEO and Founder of CB Insights, explains that headcount is a metric a LOT of people took pride in until recently. But having 500 people when you could do the work with 250 is destroying value.
And he sees a lot of startups are still in this trap based on headcount data he's dug into across thousands of data points for recent funding rounds.
Just because you have a lot of people on payroll, doesn't mean you are necessarily driving incremental value per person.
11/ Net Promoter Score (NPS)
NPS, if not done rigorously, can be very misleading.
Matt Harney (@SaaSletter), Founder of Cloud Ratings and an equity research analyst, says:
“Generally NPS is reliable... but through a few channels I've seen it weakening in rigor.”
For example: You'll see companies taking G2 Reviews, which have a structural upward bias, to impute an NPS.
Or even worse, relabeling CSAT as NPS.
Also, if you aren't winning new logos, a small base of happy, high NPS customers doesn't matter that much if your goal is to generate institutional sized returns.
(Check out Matt’s newsletter: SaaSletter)
12/ Adjusted EBITDA
I'm often overly biased to analyzing pure play SaaS companies, and forget that there are many, many more companies that actually sell physical stuff out there. You know, the stuff you can touch?
It took my friend Alex Morris (@TSOH_Investing) the founder of The Science of Hitting Investment Research, to remind me how assumptions and differences in CAPEX and D&A can be gamed.
Whenever a company cites EBITDA, dig into the assumptions to ensure it's clean.
(Check out Alex’s newsletter:
)13/ Like for Like Revenue
A hot take from @SecretCFO:
“LFL revenue is BS if you don't neutralize it for Inflation and FX impacts.
I see plenty of people reporting revenue growth on a trophy, when underlying volumes are really flying backwards"
Revenue is a function of price and volume.
You have to keep an eye on both levers, and figure out what impacts each and may skew things in your favor, masking bigger, scarier problems. Or, just don’t show any revenue at all 👇🏽
What I’ve Been Looking Forward To:
My buddy Nicolas Boucher is putting on a cohort based course on ChatGPT. And it’s specific to finance professionals. If you’re looking to harness the powers of AI for your FP&A, Finance, or Accounting job, check out the course. It takes place Tuesday April 18th at 2PM EST.
Quote I’ve Been Pondering
“Tell em you wanna direct as well, you’ll sound even less needy, they’ll salivate”
-Matthew McConaughey, Green Lights
Excellent point! Most companies I’ve worked at have 3 or 4 key metrics to optimize.
For example, a goal on customer count, one on revenue, another on churn
All three represent different levers to improve the biz
I really like how you focused on ARPU and not just total subs.
Great post - and thanks for the shout-out CJ!