Common, but costly, mistakes Product Managers make with metrics

In the movie Spiderman, Uncle Ben counsels a young Peter Parker:

It’s a quote that’s stood the test of time. But what’s lost in the interaction is that Uncle Ben was actually advising his nephew, a junior Product Manager, on the wonder, and destructive abilities, of metrics.

You see, a metric is like a weapon of mass destruction in the wrong hands. When misapplied, metrics can push teams to build things people don’t want, or deafen them to the “real” message a customer base is screaming.

And there are several common, but costly, missteps I’ve seen Product Managers make time and time again with metrics. And they’re even more likely to occur when you’re operating within a fast growing tech rock ship, and every light on the dashboard is flashing in your face.

Here are six missteps I’ve seen PM’s make, and tested solutions for fixing them.

Active user count is often put on a pedestal. And it’s not that it’s inherently wrong, but more the potential it brings for misguided application.

Most commonly the definition of “active user” is the lowest common denominator of any activity in the product, like just logging in.

But DAU (daily active users) and MAU (monthly active users) are pretty useless without supporting information or specific rules that allow you to quantify them. 

For example, if someone logs into an app every month and doesn’t use it for a specific purpose, then it’s pretty irrelevant. Arbitrary definitions of “active” can lead product teams to make the wrong decisions over time.

Active users should be those performing an action that is tied to the product’s core value proposition. For example, if you are a security product, it might be running a vulnerability test to detect bugs.

And when you go to measure active users, do it over a period of time that aligns with how often someone should realistically use your product for said value creation. If your product is a social app or music streaming service like Spotify, you may indeed want to measure activity on a daily basis. But if your product is built for Financial Reporting, it may be smarter to align the timeframe to the accounting quarter end close.

And beware that new users will mask poor retention. This frequently goes under the radar at fast growing companies. Therefore the solution is to use “user growth accounting”, which keeps an eye on the changes in new, existing, and resurrected users in a cohort period. Peep the chart from below:

Customer Lifetime Value (LTV) estimates the total amount of money you’ll get from a customer before they churn out. It effectively tells you:

But you typically need a track record of three or more years to calculate this reliably. This is because you need enough time for a customer to churn, and they may be locked into a multi year contract, or even still be in their first year.

And that’s hard for startups who haven’t been selling very long. I’ve seen lots of B2B companies assume a blanket average lifetime of 5 years across all segments. And that’s wrong.

In the absence of enough information, assumptions can make this metric’s output more optimistic forecasting rather than signal.

In the absence of long-term data, the best you might be able to do is take what churn data you do have and define lifetime period as (1 / churn rate).

And if you do have some data, you’ll want to examine both LTV and LTV to CAC by segment, or sales engine. This is because Enterprise customers usually have a higher cost to acquire than SMB customers, but sizeable life time values, driven by larger deal sizes and lower churn.

Therefore, any assumptions you use in LTV should link back to the appropriate churn benchmarks for that specific segment (see Lenny’s benchmarks by product and segment).

And if you need some rules of thumb for B2B sales engines, think about using 1.5 years for SMB, 3 years for mid market, and 6 years for enterprise.

Since NPS is effectively evaluating the entire customer experience, including every touchpoint, in many cases it can lead to conclusions about the product that are misguided. Therefore, NPS is no good for feature level evaluation.

Also, NPS, if not done rigorously, can be very misleading. You'll often see companies taking G2 Reviews, which have a structural upward bias, to impute an NPS. Or even worse, relabeling CSAT (Customer Satisfaction Score) as NPS.

What used to be such a simple, elegant metric is now being gamed, making it arguably a vanity metric.

If you are trying to compare customer satisfaction at the feature level, use CES. Throw NPS out the window.

And regardless of if you are using NPS or CES, randomize the survey population, as it’s easy to subconsciously target your happiest, most successful customers. Don’t exclusively ask for a rating after a successful purchase or an “aha” moment - that’s biased - watch where you place the question.

And as a sanity check, compare your NPS against your Net Dollar Retention - there’s a clear correlation between a customer segment with high NPS and a high NDR. If NPS is high but NDR is not top quartile, your testing is probably off or biased.

A frequently reported metric for platform companies is multi-product attach. This is the percentage of customers paying for two, three, four etc… products. 

DataDog is famous for reporting this figure (with success). 

But companies typically wait too long to roll old counts off, and don’t wait long enough to roll new counts on.

It’s very common for an attach rate to plateau, or sometimes even decline, around ~65%. And if people don’t understand the metric and see it slow down, they may panic. 

But it’s pretty natural to hit a saturation point for any product-attach count. So you should stop reporting at a predetermined saturation rate so there are no surprises, and roll it off for a new cut with more headroom and decision making significance. 

But don’t do it too soon - to address the second misstep, it’s very common for companies to rush to report a new product attach rate (e.g., moving from +5 products to +6 products) before the figure has a chance to stabilize. 

New products take some time to ramp within your customer base, and may experience some ups and downs for the first few quarters. 

And you need to make sure that customers aren’t rationalizing one of your products for another (e.g., staying at 3 total products due to budget constraints by churning a less critical product for a recently launched product).

You’ll usually want to wait until the new cut has achieved three quarters of successive increases before adding.

Many software businesses maniacally focus on ARPU (Average Revenue Per User) and how it grows over time. 

But overemphasizing growth in ARPU without digging into account retention masks a potentially leaky bucket. 

If ARPU increases, but you’re losing users at a rapid rate, that’s not healthy or sustainable growth.

Why? You’re really on a treadmill and will need to keep spending to acquire new users.

Gross account churn is the percentage of accounts that continue to spend from one period to another, regardless of the dollar amount they are spending.

In addition, you’ll want to measure what percentage of your revenue comes from different sized customers (e.g., customers over $100K, customers over $1M) to identify any customer concentration risks

I’ve worked at companies where we publicly marketed all-time users, all-time customers, all-time downloads, and all-time bug fixes. 

These stats looked great on investor splash slides, but were empty for three reasons: 

For any user graph, make sure to clearly state the measurement time period, and avoid using nonstandard cuts (e.g., “last 17 months” or “trailing five quarters”).

You should also clearly state the growth rate, whether that be month over month, quarter over quarter, or year over year. For a metric to be useful in decision making, you need to make it comparative and give it a ratio or rate to show how it is trending.

At the end of the day, a good metric is both art and science. And you can’t be blinded by the science in and of itself.

A good metric has 4 key qualities (per my product minded friend Ben Yoskovitz of Focused Chaos):

With these guidelines and guardrails, you as a Product Manager can be like Peter Parker and fight metric crime. Good luck building!