Common, but costly, mistakes Product Managers make with metrics

The myth of Active Users, NPS lies, and more

In the movie Spiderman, Uncle Ben counsels a young Peter Parker:

With great power, comes great responsibility.

It’s a quote that’s stood the test of time. But what’s lost in the interaction is that Uncle Ben was actually advising his nephew, a junior Product Manager, on the wonder, and destructive abilities, of metrics.

You see, a metric is like a weapon of mass destruction in the wrong hands. When misapplied, metrics can push teams to build things people don’t want, or deafen them to the “real” message a customer base is screaming.

And there are several common, but costly, missteps I’ve seen Product Managers make time and time again with metrics. And they’re even more likely to occur when you’re operating within a fast growing tech rock ship, and every light on the dashboard is flashing in your face.

Here are six missteps I’ve seen PM’s make, and tested solutions for fixing them.

1. The Myth of Active Users

2. Rushing to LTV Conclusions

3. The Sweet Lies NPS Tells You

4. Multi-Product Attach Panic

5. Disregarding the Quality of ARPU Growth

6. Getting Cute with Cumulative Metrics

---

The Myth of Active Users

Problem:

Active user count is often put on a pedestal. And it’s not that it’s inherently wrong, but more the potential it brings for misguided application.

Most commonly the definition of “active user” is the lowest common denominator of any activity in the product, like just logging in.

But DAU (daily active users) and MAU (monthly active users) are pretty useless without supporting information or specific rules that allow you to quantify them. 

For example, if someone logs into an app every month and doesn’t use it for a specific purpose, then it’s pretty irrelevant. Arbitrary definitions of “active” can lead product teams to make the wrong decisions over time.

Solution: 

1. Define “active” around the core value

2. Make sure the period is aligned with the problem frequency

3. Use a cohort or team based view (vs user based view) for B2B

Active users should be those performing an action that is tied to the product’s core value proposition. For example, if you are a security product, it might be running a vulnerability test to detect bugs.

And when you go to measure active users, do it over a period of time that aligns with how often someone should realistically use your product for said value creation. If your product is a social app or music streaming service like Spotify, you may indeed want to measure activity on a daily basis. But if your product is built for Financial Reporting, it may be smarter to align the timeframe to the accounting quarter end close.

And beware that new users will mask poor retention. This frequently goes under the radar at fast growing companies. Therefore the solution is to use “user growth accounting”, which keeps an eye on the changes in new, existing, and resurrected users in a cohort period. Peep the chart from

The Product-Led Geek

below:

User Growth Accounting. Source: Product Led Geek

---

Rushing to LTV Conclusions