Does AI have a gross margin problem?
Can AI overcome the gross margin doubters?
Financial operations are needlessly complex. You have to cobble together a patchwork of tools that aren’t integrated with each other, cost you time, and lead to errors.
Mercury simplifies this with banking* and software that powers all your critical financial workflows — all together within the one thing every business needs — a bank account. And with new bill pay and accounting capabilities, you can pay bills faster, stay in control of company spend, and speed up reconciliation.
*Mercury is a financial technology company, not a bank. Banking services provided by Choice Financial Group and Evolve Bank & Trust; Members FDIC.
There’s no question about AI’s revenue potential. The traction is quite literally like nothing we’ve seen before.
OpenAI surpassed $1.6 billion in ARR to close out 2023, and is probably far past $2B as I write this. Anthropic is also on a roll - forecasting to hit $850M in ARR by the close of 2024.
But if we turn our eyes away from the sun for a quick moment and peruse the rest of the P&L, you’ll find that these models aren’t cheap to run.
It begs the question: What’s the longer term profitability going to look like for AI companies? Will they be fighting gross margin headwinds in perpetuity? Will anything ever “drop to the bottom”?
What’s up there, anyways?
What’s in an AI company’s gross margin? Well, there’s the typical “cost of goods sold” - like customer support, hosting, and data feeds.
And then there’s the datacenter costs. A LOT of datacenter costs. Per the Information:
“Anthropic’s gross margin—gross profit as a percentage of revenue—was between 50% and 55% in December, according to two people with direct knowledge of the figures. That’s far lower than the average gross margin of 77% for cloud software stocks, according to Meritech Capital.”
And, depending on who you ask, it may not improve much over time: At least one major Anthropic shareholder expects the company’s long-term gross margin to be around 60%.
And that’s even before we throw in “the other stuff” that kinda sorta maybe should be contemplated:
“Notably, Anthropic’s gross margin doesn’t reflect the server costs of training AI models, which Anthropic includes in its research and development expenses. These costs can add up to as much as $100 million per model, according to Sam Altman, CEO of OpenAI.”
I asked my buddy Fred Havemeyer, a leading sell-side equity analyst from Macquarie specializing in AI and software research, about the gross margin predicament in AI:
“What is the gross margin profile of generative AI? Will it be really cool, but just a cost center for businesses? I built my own model of a tier 4 data center model, bottom up…
…You can run bigger models with like a 60% plus gross margin.
You can run some mid sized models with much higher gross margins - like 80% plus gross margins.
And if you are running really small models… purpose built generative AI models, I estimate you can run them with 90% plus gross margins… very software like gross margins.
That gave me a lot of reassurance that this is not going to be a space that is just going to be a gross margin drag on these hyper scaler software businesses; that they can run profitably…”
What Fred said made me feel a little bit better. So I started searching for precedents. OpenAI, Anthropic, or [Insert Flashy AI Company Name] wouldn’t be the first to overcome these gross margin doubts.
The first name that comes to mind is Nvidia:
But if we’re being honest, for every Nvidia there’s an Intel:
What will separate the winners from the losers in the quest for “good” gross margins are their relationships with suppliers, most notably AWS / GCP / Azure on the cloud side, and chip manufacturers, like Nvidia, AMD, and Qualcomm.
Google and Amazon have already committed billions of dollars to Anthropic. And there are a handful of jokes that could be made about Microsoft actually running / owning OpenAI.
See - I even made a meme:
And while Nvidia is head and shoulders above the rest in terms of market share, it may not be necessary to purchase the Rolls Royce of chips to deliver the same services - AI developers may find it easier to run their models on cheaper servers. Or pull a Dropbox, and move in house entirely (epic reverse migration).
Perhaps the truth lies somewhere in the middle - more akin to a Snowflake - a steady creep up into the mid to high 60’s:
So where do we go from here?
As a starting point, we probably need to better define what AI models will be used for. Software-layer AI optimizations are part of the margin equation. In Macquarie’s December 2023 Microsoft Copilot monetization report, they wrote on the cover:
“We think GenAI app economics can be improved with smaller AI models, AI-optimized hardware (e.g., Microsoft's custom AI chip), and software-layer optimizations.”
So fine-tuning AI models to your specific needs cuts down on waste, and running them on purpose-built chips helps a bunch, too.
And for all my accounting friends out there, I’d be remise to point out that some of the cost should be capitalizable (is that a word?). While it won’t help with cash flow in the near term, per Fred:
“I was interested to find software companies aren’t capitalizing and amortizing model fine tuning costs. Our checks found fine-tuning costs being recognized as in-period R&D expense rather than capitalized internal use software that would be amortized over multiple periods.”
Regardless of accounting wizardry, the pursuit of 'good' gross margins in the AI industry is not just about managing costs but about innovating and optimizing every aspect of the business model. It’s important to realize that we are still just in the early days.
Run the Numbers
Listen on Apple, Spotify, YouTube
How will AI impact your P&L? I asked my buddy Fred Havemeyer, a leading sell-side equity analyst from Macquarie specializing in AI and software research.
On this episode we cover:
The difference between the metrics the sell side uses vs operators in their day to day jobs, and how to translate between models
The art and science of valuing a company
Quantifying the productivity gains we’re seeing from AI today
The cost models underlying large language models, and what realistic long term gross margins for AI companies are
And real life use cases for finance people diving into their first large language models
Quote I’ve Been Pondering
“I’d learned over the years that when you have the energy to pick up the pace, you need to use it right away, because it doesn’t always come back”
-Ryan Hall, Run the Mile You’re In
Just one catch with comparing NVIDIA to Rolls Royce of chips then pointing out not everyone has to buy a Rolls Royce.. unlike CPUs, there's not a whole lot of competition with GPUs or NPU/TPUs yet. Someone else is going to have to open that market up, and then AI developers or pulling a Dropbox becomes viable.
Intel's too far off with ARC to give it a go.. AMD could have production capability but need more polish. There's GraphCore, Cerebras Systems but they're still small. Watching that space certainly will give a better indication as to future supply & demand to feed the OpenAI & Anthropics of the world.
Great overview CJ. I do think the gross margins will grow over time with optimizations but I think competition amongst these vendors will require them to make big investments outside of R&D to keep up. So as Moore’s law reduces the COGS there might be less differentiation requiring each to put more cash into S & M spend. I don’t know if that will be at the same scale as the R&D costs.