When the Data and the Anecdotes Disagree, the Anecdotes are Usually Right
The importance of trusting anecdotes and hunches before data
When I became an FP&A leader for the first time, I built our entire operating model from scratch. I had no context and no templates. And that blank workbook staring back at me was DAUNTING!
That’s why I’m excited to introduce our sponsor, Mercury.
Their VP of Finance, Dan Kang is sharing his personal template for building a financial forecast model.
Whether you’re preparing to fundraise, updating investors, or charting a new course for the company, you’ll likely reach a point where you’ll need to build a forecast model. It can help you align on clear business goals and understand why those goals matter.
I know I’m the metrics guy and all, but here’s something I believe: When the data and the anecdotes disagree, the anecdotes are usually right
Park Conrad, founder of Rippling, and Jeff Bezos, founder of Amazon, each shared stories about prioritizing anecdotes, and chasing down ground truths at the event level, before making decisions off of aggregate data.
Logan Bartlett was talking to Conrad about Founder Mode on his podcast and the concept of chasing down anecdotes to ground truths came up:
“The best leaders tend to solve problems by going really deep and going into the weeds. If something is wrong in customer support one approach to solving this is to look at your team and say do I have the right leader? Great, they are they working on it. They’ll figure it out.
I’ve found that doesn’t work. If the problem gets to your level wherever you sit in the organization, you’ve got to go all the way to ground.
So the way to solve that is to look at the last 50 support interactions. Or you look at the last 20 Gong calls. You go and inspect the anecdotes rather than the data. You can get to an answer much more quickly by looking at a stream of anec-data.
And once you understand the problem it’s usually pretty clear what you have to do about it. You can thrash for a long time if you are looking too top down at it and trying to operate through your direct reports and management team.
It’s equally true for directors, vps, c-level execs, as it is for founders.”
-Park Conrad, Logan Bartlett Show
Conrad explains that data is great, but solving a problem needs to start at the ground level before you start to lean on data. I mean, how do you know if it’s even being measured correctly?
Plus, if he were to just wait for updates from his direct reports, context would get lost in translation, and the feedback loop to solving the problem would be extended.
In a similar vein, Jeff Bezos told a customer support story on the Lex Friedman podcast. It began as a discussion about the importance of data based decisions at Amazon, until Bezos zagged. Instead, he spoke to prioritizing hunches and anecdotes over data:
Jeff Bezos(01:34:00) By the way, a lot of our most powerful truths turn out to be hunches, they turn out to be based on anecdotes, they’re intuition based. And sometimes you don’t even have strong data, but you may know the person well enough to trust their judgment. You may feel yourself leaning in. It may resonate with a set of anecdotes you have, and then you may be able to say, “Something about that feels right. Let’s go collect some data on that. Let’s try to see if we can actually know whether it’s right. But for now, let’s not disregard it. It feels right.”
Lex Fridman(01:33:36)You have to tell me the story of the call you made, the customer service call you made to demonstrate a point about wait times?
Jeff Bezos(01:34:00) We were going over a weekly business review and a set of documents, and I have a saying, which is when the data and the anecdotes disagree, the anecdotes are usually right. And it doesn’t mean you just slavishly go follow the anecdotes then. It means you go examine the data because it’s usually not that the data is being miscollected, it’s usually that you’re not measuring the right thing. And so of you have a bunch of customers complaining about something and at the same time, your metrics look like they shouldn’t be complaining, you should doubt the metrics.
(01:34:43) And an early example of this was we had metrics that showed that our customers were waiting, I think less than, I don’t know, 60 seconds when they called a 1-800 number to get phone customer service. The wait time was supposed to be less than 60 seconds, but we had a lot of complaints that it was longer than that. And anecdotally it seemed longer than that. I would call customer service myself. And so one day we’re in a meeting, we’re going through the WBR, the weekly business review, and we get to this metric in the deck, and the guy who leads customer service is defending the metric. And I said, “Okay, let’s call.” Picked up the phone, and I dialed the 1-800 number and called customer service, and we just waited in silence.
Lex Fridman(01:35:39) What did it turn out to be?
Jeff Bezos(01:35:40) Oh, it was really long, more than 10 minutes, I think.
Lex Fridman(01:35:42) Oh, wow.
Jeff Bezos(01:35:43) It was many minutes. And so it dramatically made the point that something was wrong with the data collection. We weren’t measuring the right thing, and that set off a whole chain of events where we started measuring it right. And that’s an example, by the way, of truth-telling is like that’s an uncomfortable thing to do, but you have to seek truth even when it’s uncomfortable, and you have to get people’s attention and they have to buy into it, and they have to get energized around really fixing things.
-Jeff Bezos on the Lex Friedman Podcast
First, it’s hilarious to picture a stone-faced Bezos, silently staring at the executive who told him the wait times were short for a full ten minutes.
TEN MINUTES!
It must have felt like a fucking eternity.
Note that Bezos didn’t say “this exec was trying to hide something…” He says that the data process was broken. And following anecdotes is an antidote to redirecting measurement. You might be tracking a data point this way because of inertia - “it’s always been measured that way” - yet the world has shifted out from underneath it.
And it’s hard to get to that point where you shift measurement if you are managing through managers. Yes, the data sets they are giving you may be statistically relevant and factually accurate. But they could be pointing a heavy weapon at the wrong target. Two degrees to the left with your howitzer of a BI tool and you think you have sub 60 second wait times.
I liken it to what Jenny Decker, former CFO of Front said on the RTN podcast - as an exec you need the ability to play both Eagle and play Mouse. This requires changing the level you operate at, zooming in and zooming out, to get the full picture.
The best executives have the ability to chase down a single customer support call, out of the thousands that may occur per day, digest what occurred, and then context shift to another decision which is based on the last twelve months of data. All of it is relevant. All of it is part of a mosaic.
The final takeaway I’ll share is that the older I get, the more I trust my gut instincts and hunches. This isn’t because I think I’m any smarter or downloading data at a faster rate than anyone else. It’s because our minds are like really slow versions of large language models, trained over time to connect the dots, subconsciously, between scattered events and data points.
In other words, the quality of my gut instincts and hunches compound as I see a higher volume and higher difficulty of business problems. If the 8th wonder of the world is compound interest, the 9th wonder of the world is compound experiences.
While I may not be able to point to why I feel a certain way in specific terms, I know it came from somewhere, or a collection of “somewheres” scattered over various context settings.
So next time you hear something that doesn’t sound right, fuck it, go check the Gong recording. It will take five minutes.
Related reading:
Beware of all green dashboards
My friend Niklas texted me “yo, Jeff Bezos is talking about metrics on the Lex Friedman podcast.”
Run the Numbers
Apple | Spotify | Youtube
Michael Linford, the COO & CFO of Affirm joined me to delve into the Buy Now, Pay Later (BNPL) business model and its rise to mainstream popularity. We discuss:
The importance of transparent, consumer-friendly financial products and how Affirm differentiates itself by avoiding the predatory practices common in traditional credit models.
How Affirm structures its fees and generates revenue.
The company’s approach to credit underwriting and the significance of short loan durations.
Key metrics like customer acquisition and lifetime value, and whether the merchant or end consumer should be the focus
Quote I’ve Been Pondering
A grandfather is talking with his grandson. The grandfather says, “In life, there are two wolves inside of us which are always at battle. One is a good wolf which represents things like kindness, bravery, and love. The other is a bad wolf which represents things like greed, hatred, and fear.”
The grandson stops and thinks about it for a second, then he looks up at his grandfather and says, “Grandfather, which one wins?”
The grandfather replies, “The one you feed.”
Loved this. We sometimes get so caught up in the numbers that we forget the meaning behind them. Thanks for sharing