Common, but costly, mistakes Product Managers make with metrics
The myth of Active Users, NPS lies, and more
In the movie Spiderman, Uncle Ben counsels a young Peter Parker:
With great power, comes great responsibility.
It’s a quote that’s stood the test of time. But what’s lost in the interaction is that Uncle Ben was actually advising his nephew, a junior Product Manager, on the wonder, and destructive abilities, of metrics.
You see, a metric is like a weapon of mass destruction in the wrong hands. When misapplied, metrics can push teams to build things people don’t want, or deafen them to the “real” message a customer base is screaming.
And there are several common, but costly, missteps I’ve seen Product Managers make time and time again with metrics. And they’re even more likely to occur when you’re operating within a fast growing tech rock ship, and every light on the dashboard is flashing in your face.
Here are six missteps I’ve seen PM’s make, and tested solutions for fixing them.
The Myth of Active Users
Rushing to LTV Conclusions
The Sweet Lies NPS Tells You
Multi-Product Attach Panic
Disregarding the Quality of ARPU Growth
Getting Cute with Cumulative Metrics
The Myth of Active Users
Problem:
Active user count is often put on a pedestal. And it’s not that it’s inherently wrong, but more the potential it brings for misguided application.
Most commonly the definition of “active user” is the lowest common denominator of any activity in the product, like just logging in.
But DAU (daily active users) and MAU (monthly active users) are pretty useless without supporting information or specific rules that allow you to quantify them.
For example, if someone logs into an app every month and doesn’t use it for a specific purpose, then it’s pretty irrelevant. Arbitrary definitions of “active” can lead product teams to make the wrong decisions over time.
Solution:
Define “active” around the core value
Make sure the period is aligned with the problem frequency
Use a cohort or team based view (vs user based view) for B2B
Active users should be those performing an action that is tied to the product’s core value proposition. For example, if you are a security product, it might be running a vulnerability test to detect bugs.
And when you go to measure active users, do it over a period of time that aligns with how often someone should realistically use your product for said value creation. If your product is a social app or music streaming service like Spotify, you may indeed want to measure activity on a daily basis. But if your product is built for Financial Reporting, it may be smarter to align the timeframe to the accounting quarter end close.
And beware that new users will mask poor retention. This frequently goes under the radar at fast growing companies. Therefore the solution is to use “user growth accounting”, which keeps an eye on the changes in new, existing, and resurrected users in a cohort period. Peep the chart from
below:
Rushing to LTV Conclusions