Strategy in the Face of Chaos
Audio companions to my writing on strategy, technology, AI, cybersecurity and building technology businesses.
Each edition explores one of my published articles through an AI-generated discussion or debate, offering another way to engage with its central ideas. These are not interviews or original podcast episodes, and the voices are not mine. The written article remains the definitive version.
This channel is currently a pilot, and the format will evolve as I learn what works.
Strategy in the Face of Chaos
The Impact of AI’s Rising Costs: What It Means for Innovation and Strategy
Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.
This pilot audio edition explores the argument behind The Impact of AI’s Rising Costs: What It Means for Innovation and Strategy.
The discussion examines how the growing cost of training frontier AI models is reshaping competition and concentrating development among a small number of well-funded organisations. It considers the constraints driving these costs, including computing infrastructure, energy, chip availability, data and latency.
It also explores the opportunities available to smaller companies and enterprises. Rather than competing directly in frontier-model development, they can use existing models, open-source tools, proprietary data and domain expertise to build specialised solutions. The episode considers why efficiency, collaboration and strategic focus may become as important as scale in determining who creates value from AI.
This is an AI-generated discussion based on my published article. It offers another way to engage with the ideas, but it is not an interview or a recording of me. The written article remains the definitive version.
You know, there's this um this really comforting, almost invisible assumption that's just built into the very foundation of the software industry.
SPEAKER_00Aaron Powell Oh, yeah, the classic Silicon Valley playbook.
SPEAKER_03Exactly. It's the idea that you write a piece of code once and then deploying it a million times or, you know, a billion times, it costs practically nothing. Aaron Powell Right.
SPEAKER_00The marginal cost basically approaches zero.
SPEAKER_03Yeah. And that assumption is the engine that built the modern tech industry, really. But then you step into the world of frontier artificial intelligence, and suddenly that whole economic rule book just gets thrown completely out of the window.
SPEAKER_00Aaron Powell It really does. We are looking at a landscape where the costs aren't shrinking as the technology matures. They're actually exploding.
SPEAKER_03Which is crazy to think about.
SPEAKER_00And that brings us to Victor Holman's recent piece, which is titled The Impact of AI's Rising Costs, What It Means for Innovation and Strategy. It completely upends that whole zero marginal cost illusion.
SPEAKER_03Absolutely. Holman's analysis is heavily anchored by some um, frankly, staggering data from Epoch AI. It really forces us to look past the marketing hype. Because everyone is just focused on the hype right now, right?
SPEAKER_00Yeah, but we are hitting hard physical and financial constraints that are actively reshaping the industry right under our feet.
SPEAKER_03Aaron Powell, which is exactly why we are doing this today. If you are a technology or business leader listening right now, we are treating this deep dive as a rigorous executive audio briefing specifically for you.
SPEAKER_00Aaron Powell We want to get into the real numbers and the physical realities.
SPEAKER_03Aaron Powell Right. We're going to break down the true cost of AI scaling, the four major physical bottlenecks threatening progress, and most importantly, the practical strategic implications for your enterprise.
SPEAKER_00Aaron Powell Because you definitely need a strategy for this.
SPEAKER_03Okay. Let's unpack this by starting with the sheer scale of the financial reality. Before we figure out what's causing the problem, we need to understand the trajectory of the spending.
SPEAKER_00Aaron Powell And the numbers are definitely the best place to start because they just immediately shatter that illusion of software getting cheaper. Let's look at the historical data Holzmann lays out.
SPEAKER_04Okay.
SPEAKER_00So back in 2017, training one of the most advanced AI models cost around $600,000.
SPEAKER_03Aaron Powell Which, I mean, that's a huge sum for a basic research project, but it's pocket change for a tech giant.
SPEAKER_00Aaron Powell Exactly. It's nothing to that. But fast forward just six years to 2023, training OpenAI's GPT-4 cost an estimated $40 million.
SPEAKER_01Wow.
SPEAKER_00And we have to be very precise here. That $40 million represents only the amortized costs.
SPEAKER_03Now wait, let me pause you there because we hear that term thrown around a lot in cloud computing, but I think it can be misleading. When you say the amortized cost was $40 million, what does that actually mean for the company writing the check?
SPEAKER_00Yeah, this is a crucial factual caveat for any leader who's thinking about infrastructure. Amortization means spreading out the cost of a massive asset over its useful life.
SPEAKER_01Right.
SPEAKER_00So that $40 million essentially represents the fractional wear and tear depreciation of the hardware, plus the energy used specifically during the months it took to run that one training cycle.
SPEAKER_03Okay, so it is not the upfront cost.
SPEAKER_00Not at all. To actually perform that training, you have to possess the hardware first. For GPT-4, that required an estimated $800 million upfront hardware acquisition.
SPEAKER_03Aaron Powell Just to build the supercomputer that could run the training.
SPEAKER_00Exactly. You are just swiping a credit card to rent a cloud server for a weekend. The Epic AI data breaks this down beautifully. Hardware accounts for about 44% of those amortized costs over time.
SPEAKER_02Okay, and what about the rest of it?
SPEAKER_00Well, the RD staff, so the highly specialized researchers, their salaries, their equity, that makes up a massive 29 to 49 percent of the ongoing expenses.
SPEAKER_03And the scary part is we are nowhere near the ceiling on this. Holman points out that training costs for the most compute-intensive models, you know, these frontier models, they're growing at roughly 2.4 times per year.
SPEAKER_00Every single year.
SPEAKER_03Yeah. If that compound growth holds, Epic AI projects that by 2027, training a single frontier model will exceed $1 billion.
SPEAKER_00Aaron Powell One billion dollars for one software run?
SPEAKER_03I like to compare this to aviation. It's not just that we're trying to build a faster airplane. It's like we're suddenly trying to build an airplane the size of a city and realizing we don't have enough metal or a runway long enough to actually test the thing.
SPEAKER_00Aaron Powell That's a great analogy. What's fascinating here is that these skyrocketing costs aren't just the result of bad code or corporate inefficiency.
SPEAKER_03Aaron Powell Right. They aren't just wasting money.
SPEAKER_00Aaron Powell No, they are driven by physical realities. The industry is hitting a wall, or more accurately, four specific physical bottlenecks that make scaling an exponentially harder problem.
SPEAKER_03Let's get into those bottlenecks. Because I think a lot of people just assume, well, computers always get faster, Moore's law will save us, right?
SPEAKER_00Aaron Powell Yeah, that's a very common assumption.
SPEAKER_03Aaron Powell But the first bottleneck actually has nothing to do with processing power, it's electrical power. I mean, AI is literally eating the electrical grid.
SPEAKER_00Aaron Powell It really is. To give you some context on this, US data centers currently use about 20 gigawatts of power in total.
SPEAKER_01Okay.
SPEAKER_00But as we scale these models up to try and achieve more advanced reasoning, the power demands become astronomical. Epoch AI projects that by 2030, a single frontier training run could require up to six gigawatts.
SPEAKER_03Six gigawatts. Wait, just for some real-world perspective, isn't a standard nuclear reactor output roughly one gigawatt?
SPEAKER_00Yes. You are talking about the equivalent of a massive multi-reactor power plant, or basically a mid-sized city's entire power draw.
SPEAKER_03Just for one training run.
SPEAKER_00Yeah. Imagine a company having to secure the energy equivalent of Seattle just to plug in the computers required to train one software model. And you can't just, you know, spawn a power plant or a new transmission line overnight.
SPEAKER_03Right. And you see this happening already. Data center developers in places like Northern Virginia or Ireland, they are fighting massive battles with local utility boards because the local grids physically cannot handle the requested load.
SPEAKER_00Yeah, the infrastructure just isn't there.
SPEAKER_03So you might have the billions to buy the servers, but you literally cannot get the electricity to turn them on.
SPEAKER_00Exactly. And even if you magically solve the power issue, you run face first into bottleneck number two, which is chip manufacturing capacity.
SPEAKER_03Now we all know GPUs, those graphics processing units, are the engines of AI. The research mentions that by 2030, a massive frontier training run, which they call a 9E29 flop run, would require 100 million H100 equivalent GPUs. First of all, translate that jargon for us. What is a 9E29 flop run? And why is the H100 the benchmark?
SPEAKER_00Aaron Powell Sure. So flop stands for floating point operation. It is essentially the most basic mathematical calculation a computer can do.
SPEAKER_01Okay.
SPEAKER_009E29 is just scientific notation. It's a 9 with 29 zeros after it. It's a measure of the total unfathomable amount of math required to train a model of that size.
SPEAKER_02Got it. And the H100?
SPEAKER_00The NVIDIA H100 is currently the gold standard chip for doing that specific kind of math incredibly fast, but needing 100 million of them presents an impossible manufacturing hurdle.
SPEAKER_03Aaron Powell Wait, so the bottleneck isn't actually making the silicon chip itself, because I always assumed the shortage was about the foundries trying to print the microscopic transistors on the silicon wafers.
SPEAKER_00Aaron Powell Yeah, that is a very common misconception. Printing the silicon is hard, but it's not the primary bottleneck. The real choke point actually lies in advanced packaging technologies.
SPEAKER_02Advanced packaging.
SPEAKER_00Specifically things like COWAS, which stands for chip on wafer on substrate and high bandwidth memory, or HBM.
SPEAKER_02Okay, what is advanced packaging? Why is that holding everything up?
SPEAKER_00Aaron Powell Think of it like building a hyper advanced Formula One car. Manufacturing the raw engine block, you know, the silicon is one thing. But seamlessly connecting that engine to the transmission, the fuel lines, and the electronics so they communicate flawlessly in microseconds, that's the packaging.
SPEAKER_04Oh, okay.
SPEAKER_00These AI chips require incredibly complex three-dimensional stacking of memory modules directly on top of or right next to the processors. The data has to move back and forth almost instantaneously.
SPEAKER_03Aaron Powell So it's essentially a plumbing issue. The pipes between the memory and the brain have to be microscopic and perfectly assembled.
SPEAKER_00Precisely. And the facilities in the world that have the precision equipment to actually manufacture and assemble those microscopic high bandwidth connections are extremely limited.
SPEAKER_03Aaron Powell You can't just throw up a new factory for that.
SPEAKER_00No, you can't just build a new advanced packaging plant in a few months. So again, even if you have a billion dollars in cash, you are stuck in a supply chain traffic jam.
SPEAKER_03Which brings us to the third bottleneck, which might actually be the most counterintuitive one. We are running out of data.
SPEAKER_00Yes. Data scarcity.
SPEAKER_03These large models learn by predicting the next word, which means they are incredibly data hungry. And Holman's article points out there are an estimated 500 trillion tokens of web text available to train on.
SPEAKER_00And just to clarify for everyone, a token in AI isn't necessarily a whole word. It's often a chunk of characters. But essentially, those 500 trillion tokens represent the sum total of high-quality human knowledge written on the public internet.
SPEAKER_03Aaron Powell But we are chewing through that finite resource incredibly fast. Now let me challenge you on this because I hear this proposed as a solution all the time. Sure. If we run out of natural human-written data on the web, can't these companies just use their existing AI to generate massive amounts of synthetic data and then train the next model on that? It seems like an infinite loop of free data.
SPEAKER_00Aaron Powell It definitely sounds like an elegant workaround, but Holman highlights a very strict factual caveat here. Synthetic data is incredibly risky.
SPEAKER_03Really? Why?
SPEAKER_00Because if you train an AI primarily on data generated by an AI, you trigger a mathematical degradation known as model collapse.
SPEAKER_03Model collapse. What is actually happening under the hood there?
SPEAKER_00Think of it like taking a highly detailed photograph, making a photocopy of it, and then making a photocopy of the photocopy.
SPEAKER_03Oh, so it just gets worse and worse.
SPEAKER_00Exactly. With each iteration, the subtle nuances, the sharp edges, and the complex realities are lost. Small errors or biases in the AI's output are fed back into the next model as absolute truth, which just magnifies the errors.
SPEAKER_04So it just spirals.
SPEAKER_00Eventually. The model forgets the underlying complex distribution of real human language, and the output devolves into a repetitive, blurry mess.
SPEAKER_03So it essentially becomes an echo chamber that drives itself insane.
SPEAKER_00That's a great way to put it. Furthermore, generating high-quality synthetic data that doesn't immediately cause collapse requires incredibly complex filtering.
SPEAKER_03Which takes compute power.
SPEAKER_00Exactly. It introduces massive computational overhead. You end up spending a fortune on expensive compute just to create the data to do the training. It is a highly inefficient, costly workaround, not a magic bullet.
SPEAKER_03Okay. So we're running out of power, we're stuck in a chip packaging traffic jam, and we are literally running out of original human text. That leaves the fourth bottleneck.
SPEAKER_00Right. The latency wall. This is the fundamental speed limit of distributed computing systems.
SPEAKER_03Aaron Powell When you say speed limit, you mean the literal speed of light, right?
SPEAKER_00Exactly. Right. When a model grows to a trillion parameters, you cannot fit it onto a single chip or even a single server rack.
SPEAKER_02Right. It's just too big.
SPEAKER_00It has to be distributed across tens of thousands of GPUs spread across a massive data center. But for the model to learn, those GPUs have to constantly talk to each other, updating their weights and parameters in real time.
SPEAKER_03So instead of the city-sized airplane analogy people always use, it's really more like trying to coordinate a symphony of like a hundred thousand musicians, but they are spread out across the entire state of Texas.
SPEAKER_00Exactly.
SPEAKER_03No matter how fast your fiber optic cables are, the signal takes physical time to travel from the violins in Dallas to the percussion in Austin.
SPEAKER_00That is a perfect metaphor. As you add more GPUs to try and brute force the training faster, the communication delays between them magnify.
SPEAKER_02The lag gets worse.
SPEAKER_00The musicians spend more time waiting to hear the beat than they do actually playing notes. In computing terms, you start spending more time moving data across the network than crunching numbers on the silicon.
SPEAKER_03Well, can't they just have each GPU process a bigger chunk of data before they talk to each other?
SPEAKER_00They do try that. It's called increasing the batch size. But if the musicians play too many notes before checking in with a conductor, the whole symphony goes out of tune. Right. Pushing batch sizes higher eventually hits strict diminishing returns where the model stops learning efficiently. Without radical, currently unproven innovations in network topies, this latency wall puts a hard physical cap on how fast we can train these massive systems.
SPEAKER_03Here's where it gets really interesting. Because having established these massive costs and these unyielding physical limits, we have to look at the market reality.
SPEAKER_00Yeah, the business side of this is wild.
SPEAKER_03Right. If the constraints of power, custom packaging, and billion-dollar capital favor only the most well-funded organizations, who survives this?
SPEAKER_00The data makes it undeniable. We are witnessing a severe market bifurcation. The era of a couple of guys in a garage building a frontier foundational model is completely over.
SPEAKER_03It's just a rich man's game now.
SPEAKER_00Only big tech giants, you know, your Microsoft's, Googles, Amazon's, Metas, and perhaps government-backed labs can actually afford the $1 billion table stakes.
SPEAKER_04It's an exclusive club, and the velvet rope is made of gigawatts and co-ops packaging.
SPEAKER_00Right. And this dynamic rapidly accelerates oligopolistic control over foundational models. When the barrier to entry is a billion dollars in capital and priority access to limited chip manufacturing, it severely restricts broad competition at the absolute bleeding edge.
SPEAKER_03Which raises an important question regarding what happens to the rest of the ecosystem. I mean, if only three or four companies control the foundational intelligence layers of the future, what does that mean for market dynamics?
SPEAKER_00It's a huge concern.
SPEAKER_03What does it mean for regulatory scrutiny over massive environmental sustainability issues like those six gigawatt data centers?
SPEAKER_00It means those few companies are going to face unprecedented regulatory pressure regarding their grid usage and market dominance. But more importantly for our listeners, it means the enterprise strategy has to change.
SPEAKER_03Right, because you can't just copy them.
SPEAKER_00As a business leader, you cannot play their game of brute force scale. You have to play a completely different one.
SPEAKER_03And that brings us to the most crucial part of this deep dive, the executive playbook. Let's pivot directly to you, the listener.
SPEAKER_00This is where it gets practical.
SPEAKER_03If you are running an enterprise, you obviously cannot authorize a $1 billion capital expenditure to build a frontier model. So how do you actually compete? Do you just surrender and pay whatever the big tech monopolies demand?
SPEAKER_00If we connect this to the bigger picture, the answer is absolutely not. But you have to be highly strategic. First, let's talk about pricing.
SPEAKER_01Okay.
SPEAKER_00Howman warns that you should absolutely expect these massive training costs to be passed down to your enterprise through tiered or usage-based pricing on APIs.
SPEAKER_03Aaron Powell So our software bills are definitely going up.
SPEAKER_00Yes. But you have a buffer. You have to understand the fundamental difference between training a model and inference.
SPEAKER_03Right. Training versus inference.
SPEAKER_00Training is the billion-dollar process of teaching the AI. Inference is the act of actually using the finished model to get an answer or generate a report.
SPEAKER_03Okay. That distinction is huge.
SPEAKER_00Enterprise AI services are heavily focused on inference. The big providers are relentlessly optimizing their inference infrastructure. And the economies of scale there will actually help shield enterprises from the most extreme price hikes.
SPEAKER_03Okay. So running the model is infinitely cheaper than building it. But how do we gain a competitive advantage if my company and my competitor's company are both just buying the exact same inference API from the exact same tech giant?
SPEAKER_00Aaron Powell This is where Holman's analysis highlights the massive opportunity for smaller and mid-sized companies. You do not need a frontier model to win in business. You need a model that solves your specific problem perfectly.
SPEAKER_03I really want to emphasize that point. It's not about having the smartest AI in the world. It's about having the most useful AI for your specific domain.
SPEAKER_00Exactly.
SPEAKER_03So let's break down how we actually do that based on the strategic actions outlined in the sources. The first major move is fine-tuning open source models rather than relying solely on proprietary frontier models.
SPEAKER_00Right. Instead of paying a massive premium for a generalist model to do everything, you can take a highly capable open source model like Bloom or Llama and fine-tune it. Which is much cheaper, right? It costs a tiny fraction of training a model from scratch, but it allows you to adapt a very smart baseline model to the specific nuances of your industry.
SPEAKER_03Which perfectly sets up the next critical piece, which is proprietary data. Holman refers to this as the ultimate moat for a business.
SPEAKER_00And he is entirely right. Think about it. A massive frontier model knows a little bit about everything. It's a generalist.
SPEAKER_01Yeah.
SPEAKER_00But if you take a smaller open source model and fine-tune it exclusively on your enterprise's highly specific proprietary data, say a decade's worth of specialized internal legal analysis or your firm's unique supply chain logistics, that specialized model will frequently outperform the billion-dollar frontier model on that specific task.
SPEAKER_03Because the generalist model might know how to write a generic legal contract, but your fine-tuned model knows exactly how your lawyers negotiate your specific contracts.
SPEAKER_00Exactly.
SPEAKER_03Your proprietary data is your ultimate competitive advantage because it is the one thing the big tech monopolies cannot scrape from the public web.
SPEAKER_00Precisely. Now, even fine-tuning takes compute power, which we know is expensive. This is why the research highlights several fascinating technical innovations categorized as efficiency techniques.
SPEAKER_04Okay, tell me more about those.
SPEAKER_00These are democratizing AI access. And as a leader, you really need to understand how they work.
SPEAKER_03Let's spend some time on this because the jargon here gets thick. Let's start with LoRa, which stands for low rank adaptation. How does that actually allow enterprises to train models cheaper?
SPEAKER_00The best way to understand LoRa is to imagine a massive multivolume medical textbook. In traditional fine-tuning, to add new information, you would essentially have to rewrite and reprint the entire textbook.
SPEAKER_01That sounds computationally exhausting.
SPEAKER_00It is. But LoRa mathematically freezes the original model, the textbook, so you don't even touch it. Instead, it trains a tiny separate layer of data that acts like a set of sticky notes placed on specific pages of the textbook.
SPEAKER_01Oh wow. So you were only spending compute power to write the sticky notes, not rewrite the whole book.
SPEAKER_00Exactly. You get the updated knowledge at a fraction of the computational cost of memory. Another vital technique is model distillation.
SPEAKER_03Distillation. So we are boiling something down. Right.
SPEAKER_00You take a massive, expensive teacher model, and you have it train a much smaller, cheaper student model. But the trick is how it trains it.
SPEAKER_04How so?
SPEAKER_00The teacher doesn't just give the student the final answer. It outputs its step-by-step mathematical reasoning, the probabilities it considered. The small student model learns to mimic the teacher's sophisticated logic patterns directly.
SPEAKER_03That is brilliant.
SPEAKER_00It's like learning the formula instead of just memorizing the answer key. Exactly. Achieving high performance without ever needing to process the petabytes of original training data.
SPEAKER_03And the final efficiency technique mentioned is quantization. What exactly are we mathematically compressing here?
SPEAKER_00At its core, an AI model is just a massive collection of numbers called weights. Traditionally, these numbers are stored in high precision formats, like 32-bit floating point numbers. They take up a lot of memory. Right. Quantization is the mathematical process of rounding those complex numbers down into lower precision formats, like 8-bit integers.
SPEAKER_03Aaron Powell So it's like taking a high-resolution image and dropping the file size, but does it make the model dumber?
SPEAKER_00Counterintuitively, very little accuracy is lost. But by shrinking the mathematical footprint, the model takes up drastically less memory and can run much faster on significantly cheaper older generation hardware.
SPEAKER_03So you're getting like 95% of the performance for 10% of the hardware cost?
SPEAKER_00Exactly.
SPEAKER_03That completely changes the ROI calculation for an enterprise. And to run these techniques, you don't even need to buy the servers. The playbook heavily emphasizes utilizing cloud and collaboration ecosystems.
SPEAKER_00Right. Platforms like AWS SageMaker, Google Vertex AI, and incredibly vibrant open source hubs like Hugging Face allow you to avoid heavy upfront capital investments.
SPEAKER_03You just rent what you need.
SPEAKER_00You rent the exact infrastructure you need when you need it and leverage the collective RD of the global open source community.
SPEAKER_03So what does this all mean? If we synthesize everything we've unpacked today from Victor Holman's analysis and the epoch AI data, the core takeaway for you as a leader is this: the era of cheap, infinite AI scaling is over.
SPEAKER_00It really is.
SPEAKER_03The physical world, the reality of power grids, copper wire, and silicon manufacturing, it has finally caught up with the software world.
SPEAKER_00The physical constraints are very real. Power, chips, data scarcity, and the latency wall are permanently reshaping the competitive landscape.
SPEAKER_03But the future doesn't just belong to the trillion-dollar companies building the biggest models. In fact, for enterprises, the future belongs to those who prioritize efficiency, specialized domain knowledge, and the strategic application of proprietary data.
SPEAKER_00Absolutely.
SPEAKER_03It's about being smart, highly specialized, and efficient, rather than just relying on brute force scale.
SPEAKER_00I want to leave you with a final thought to ponder, building directly on the realities Holman outlined today.
SPEAKER_04Okay, let's hear it.
SPEAKER_00We've spent this entire briefing talking about massive physical and financial hurdles, the strain on power grids, the microscopic chip bottlenecks, the billion-dollar price tags. But consider this. What if the ultimate constraint on frontier AI isn't technical and it isn't physical?
SPEAKER_04What do you mean?
SPEAKER_00What if the ultimate constraint is actually economic adoption?
SPEAKER_02You mean the willingness of the market to actually open its wallet?
SPEAKER_00Exactly. Think about it from the perspective of the broader economy. If a big tech company successfully spends $1 billion, or eventually $5 billion to train the next frontier model, they have to charge a massive premium to recoup that monumental cost. Aaron Powell Right.
SPEAKER_03They have to make their money back somehow.
SPEAKER_00Aaron Powell But if enterprises like yours follow the playbook we just discussed, if you figure out how to solve your specific high-value business problems using cheaper, fine-tuned, highly efficient open source models, will the market actually be willing to pay the massive premiums required to sustain those big tech monopolies?
SPEAKER_03Aaron Powell Oh, wow. That is a fascinating flip of the script. If the buyers don't need the billion-dollar model, the whole financial house of cards shakes.
SPEAKER_00Right. If the vast majority of real-world business value can be extracted from smaller, specialized models, the multi-billion dollar frontier models might become incredible scientific achievements that simply struggle to find a profitable business case.
SPEAKER_02That makes a lot of sense.
SPEAKER_00So it leads us with this question to explore on your own who really holds the true power in the AI arms race? Is it the oligopoly building the trillion parameter models, or is it the enterprise leaders, like you, who are deciding what to actually buy?
SPEAKER_03The zero marginal cost illusion might be dead, but the power of the strategic buyer is very much alive. A huge thank you to Victor Holman and the researchers at Epoch AI for the phenomenal data that drove today's briefing. And thank you for joining us on this deep dive. We'll see you next time.