Strategy in the Face of Chaos
Audio companions to my writing on strategy, technology, AI, cybersecurity and building technology businesses.
Each edition explores one of my published articles through an AI-generated discussion or debate, offering another way to engage with its central ideas. These are not interviews or original podcast episodes, and the voices are not mine. The written article remains the definitive version.
This channel is currently a pilot, and the format will evolve as I learn what works.
Strategy in the Face of Chaos
Breaking Through the Performance Plateau: Rethinking LLM Strategies for Enterprises
Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.
This pilot audio edition explores the argument behind Breaking Through the Performance Plateau: Rethinking LLM Strategies for Enterprises.
The discussion examines whether large language models are reaching a performance plateau. Newer models require increasing amounts of computing power, energy and investment, while delivering more incremental improvements. At the same time, the capabilities of leading models are converging, making cost, integration, governance and compliance more important sources of differentiation.
It explores the potential value of a multi-LLM strategy. Different models can be matched to different tasks, queries can be routed according to cost and complexity, and sensitive workloads can be handled through private or self-hosted systems. This can improve specialisation, flexibility and resilience while reducing dependence on a single provider.
The episode also considers the costs and operational complexity of an LLM mesh. For large enterprises with diverse, high-stakes use cases, the benefits may justify the investment. For organisations with simpler requirements, a smaller number of fine-tuned models or open-source alternatives may provide a more practical and economical approach.
The central argument is that AI progress is not necessarily slowing down. It is changing direction, from building ever-larger models towards applying existing models more intelligently.
This is an AI-generated discussion based on my published article. It offers another way to engage with the ideas, but it is not an interview or a recording of me. The written article remains the definitive version.
So if you listen to like the mainstream headlines right now, you would honestly think we are maybe, I don't know, six months away from some all-knowing godlike AI that's just going to seamlessly run the entire planet.
SPEAKER_01Right. Yeah. The hype is completely out of control. Trevor Burrus, Jr.
SPEAKER_00It really is. But um, if you actually talk to the engineers who are building these systems, they are quietly hitting a massive, incredibly expensive brick wall.
SPEAKER_01Aaron Powell Exactly. I mean, the disconnect between the public hype cycle and the internal reality of the industry right now is it's wider than it has ever been. The people actually writing the code and managing these massive data centers, they are signaling a major fundamental shift in how this technology progresses from here.
SPEAKER_00Aaron Powell And that is exactly what we're unpacking for you today. So our mission for this deep dive is to give you a rigorous, totally hype-free executive briefing on the actual state of enterprise AI strategy. Trevor Burrus, Jr.
SPEAKER_01Which is so needed right now.
SPEAKER_00Seriously. We are pulling our insights today from a really brilliant, sobering piece by Victor Holman. It's titled Breaking Through the Performance Plateau. And we're going to look strictly at what the data and the actual industry leaders are telling us, right? Yeah.
SPEAKER_01No fluff.
SPEAKER_00No fluff. So the core thesis we need to tackle today is basically this: the era of simply throwing, you know, billions of dollars at the biggest, shiniest new AI model to solve all your business problems, that era is officially over.
SPEAKER_01It's done. I mean, Hallman makes a really compelling case that the industry has hit this structural performance plateau.
SPEAKER_00Aaron Powell Okay, unpack that a bit. What does that actually mean?
SPEAKER_01Aaron Powell Well, for the last few years, the entire development playbook was built on what we call scaling laws. And the assumption was just pure brute force. You know, like if you build a model with 10 times the parameters and you feed it 10 times the training data and you burn 10 times the electricity.
SPEAKER_00The model just automatically gets exponentially smarter.
SPEAKER_01Exactly. But that math is literally breaking it down right now. We're seeing massive diminishing returns. The gains in raw intelligence are becoming, well, marginal, but the financial and computational costs are becoming totally unsustainable.
SPEAKER_00It's just too expensive.
SPEAKER_01Way too expensive. So the new enterprise playbook has to abandon that brute force scaling in favor of uh optimization, specialization, and hybrid architectures.
SPEAKER_00Aaron Powell I want to pause on the stagnation thesis because you know calling it a plateau makes it sound like some theoretical academic debate.
SPEAKER_01Aaron Powell Oh, it's not.
SPEAKER_00Right. Victor Holman brings the receipts. This isn't some prediction for five years from now.
SPEAKER_01Yeah.
SPEAKER_00It is actively happening this quarter with the biggest players on Earth.
SPEAKER_01Aaron Powell The evidence is piling up so fast. And it really exposes the hard limits of those scaling laws we just talked about. I mean, look at Google.
SPEAKER_00Yeah. Gemini.
SPEAKER_01Right. They have Gemini 2.0 slated for release this December. But there are prominent rumors inside the industry right now suggesting it might not deliver that massive leap in reasoning capabilities that everyone originally projected.
SPEAKER_00Aaron Powell, which is huge news.
SPEAKER_01It is. Or look at Anthropic. They literally paused the development of their massive flagship model, Opus 3.5.
SPEAKER_00Wait, they paused it.
SPEAKER_01Yeah, they hit pause. And instead, they shifted their engineering focus to a smaller, more efficient model, Sonnet 3.5.
SPEAKER_00Oh wow.
SPEAKER_01And they dedicated their resources to agent-based AI rather than just, you know, raw brain power.
SPEAKER_00Aaron Ross Powell That is such a massive signal. And even Bill Gase recently went on the record saying the leap from GPT-4 to GPT-5 is likely going to be much smaller than the giant leap we saw from GPT-3 to 4.
SPEAKER_01Aaron Powell Right. The curve is flattening.
SPEAKER_00Aaron Powell But to really understand why this is happening, I feel like we need to look under the hood. Like, why does adding more parameters suddenly cause the whole machine to grind to a halt? What is physically preventing these companies from just, I don't know, building a bigger data center?
SPEAKER_01Aaron Powell Well, it comes down to the unforgiving physics of hardware and honestly our energy grids.
SPEAKER_00Aaron Powell The power grid.
SPEAKER_01Yeah. Because to train a massive foundational model, you are stringing together tens of thousands of GPUs, but those GPUs have hard memory limits. At a certain scale, the sheer volume of data that has to move between the chips creates a really severe bandwidth bottleneck. The processors basically end up sitting idle, just waiting for data to arrive across the network rather than actually computing anything.
SPEAKER_00So they're just wasting time and power.
SPEAKER_01Exactly. And on top of that, you have the power grid constraints. We're talking about training runs that consume the electricity of a small city.
SPEAKER_00That is insane to think about.
SPEAKER_01It really is. You cannot just plug in another 20,000 GPUs without literally having to build a new power plant to support them. Wow.
SPEAKER_00But beyond the physical hardware, Hallman points out there is a profound mathematical issue here too, right? He mentions a severe lack of model interpretability as these systems scale up.
SPEAKER_01Yeah. This is a huge problem for enterprise adoption. When you scale a neural network to like a trillion parameters, which are essentially the artificial synapses of the model, it becomes this completely impenetrable black box.
SPEAKER_00Aaron Powell Right. You can't see how it's thinking.
SPEAKER_01Exactly. If it hallucinates a fact or makes a catastrophic error in, say, a financial summary, the engineers often literally cannot pinpoint exactly why it made that specific connection.
SPEAKER_00Aaron Powell That's terrifying for a business.
SPEAKER_01It is. When you scale up the size of the brain, you also scale up the complexity of the errors, and that makes it exponentially harder to debug and ensure the kind of perfect reliability that, you know, enterprise use cases demand.
SPEAKER_00Aaron Powell Which honestly brings us to the ultimate irony of this whole situation.
SPEAKER_01Yeah.
SPEAKER_00Look at Ilyas Utzkiver.
SPEAKER_01Oh, yeah.
SPEAKER_00Right. He was one of the co-founders of OpenAI, deeply instrumental in pushing the philosophy that bigger is always better.
SPEAKER_01The pioneer of scaling.
SPEAKER_00Exactly. But he recently left to start his own venture, Safe Superintelligence. And he explicitly stated, quote, it's not just about scaling anymore, it's about what we're scaling.
SPEAKER_01That quote is so telling.
SPEAKER_00It really is. He's suggesting the industry actually needs to return to the discovery phase. I mean, when the guy who pioneered massive scaling is telling the market we need to rethink the entire approach, you really have to pay attention.
SPEAKER_01You absolutely do. It completely upbends the core business assumption of the last two years.
SPEAKER_00How so?
SPEAKER_01Well, every tech leader secretly assumed that if they just waited six months, a new, infinitely smarter foundational model would just drop from the sky and magically solve all their complex workflow problems.
SPEAKER_00Trevor Burrus, Jr. Right, like an easy button.
SPEAKER_01Yeah. But if the massive models are stalling out, we have to look at how that actually impacts the market itself.
SPEAKER_00Aaron Ross Powell Well, if they all hit the same physical and mathematical ceiling, they inevitably start to look exactly like one another.
SPEAKER_01Aaron Powell Exactly. Trevor Burrus, Jr.
SPEAKER_00This is the commoditization of generative AI. We are seeing a rapid, undeniable convergence in capabilities across OpenAI, Google, Anthropic, cohere all of them.
SPEAKER_01Aaron Powell Yeah, because they're all scraping essentially the exact same internet data and using very similar underlying transformer architectures. When the recipe and the ingredients are identical, the final output is going to be incredibly similar.
SPEAKER_00Aaron Powell It totally reminds me of the early days of smartphones. You know, 15 years ago, every phone had a wildly different operating system, crazy physical keyboards, totally different form factors.
SPEAKER_01Oh, yeah, the old blackberries and flip phones.
SPEAKER_00Right. You bought a phone because it could do something completely unique that the others just couldn't. But today, I mean, a high-end Android and an iPhone basically offer the exact same core features in slightly different fonts.
SPEAKER_01Yeah, you're buying into a brand ecosystem, not a vastly superior piece of hardware.
SPEAKER_00Exactly. And Victor Holman uses a couple of brilliant analogies from the text to make this visceral. He actually compares these big name LLMs to different brands of bottled water.
SPEAKER_01I love that analogy because for common enterprise use cases like summarizing a 10-page document or drafting a routine email or analyzing basic sentiment in a customer review, the outputs from a GBT-4.0 and a Claude 3.5 sonnet are virtually indistinguishable.
SPEAKER_00You are literally just choosing between Dasani and Aquafina.
SPEAKER_01Exactly. It's the same water.
SPEAKER_00He also compares them to electricity providers, which I thought was really sharp. When you plug your laptop into the wall at the office, you do not care which regional power plant generated those specific electrons. You just want the provider that gives you the best reliability and the cheapest kilowatt hour.
SPEAKER_01And the strategic implication of this commoditization is massive for anyone leading an enterprise transformation right now. It means you no longer need to exhaust your entire IT budget chasing the absolute latest, most expensive model update just to stay competitive.
SPEAKER_00The battlefield has shifted.
SPEAKER_01Fundamentally shifted away from raw AI capability.
SPEAKER_00Okay, so if I'm listening to this and I'm a business leader and I'm no longer competing on having the quote-unquote smartest AI in the room, what is my new differentiator? What am I actually competing on now?
SPEAKER_01Aaron Powell You are competing on cost efficiency, how seamlessly the model integrates into your existing proprietary systems and crucially strong who has the smartest AI. It is entirely about who uses their AI the smartest within their specific operational context.
SPEAKER_00Okay, but this raises a glaring contradiction in Holman's argument, and I really need you to help me square this circle.
SPEAKER_01Sure, lay it on me.
SPEAKER_00If all these big models are becoming identical commodities, if they're just different brands of bottled water, why does he then spend a significant portion of the article suggesting businesses adopt an architecture of multiple specialized LLMs?
SPEAKER_01The LLM mesh.
SPEAKER_00Right, the LLM mesh. If they're all the same, why on earth would I build a complex mesh of different waters? That makes no sense.
SPEAKER_01Yeah, I get why that sounds contradictory, but that contradiction disappears when we separate generalized models from specialized models. Okay. So a massive generalized model is built to do absolutely everything. It can draft a complex legal contract, write Python code, or answer a basic customer support query about store hours.
SPEAKER_00The jack of all trades.
SPEAKER_01Exactly. But because it tries to do everything, it carries immense computational overhead. It is inherently inefficient compared to a smaller specialized model that has been fine-tuned to do just one specific task perfectly.
SPEAKER_00Ah, I see. So the LLI Mesh isn't about collecting different brands of generalized models, it's about task specialization.
SPEAKER_01Task specialization, architectural flexibility, and cost control. You build a network of models tailored to specific functions within your business.
SPEAKER_00Holman has another great analogy for this from the text. He calls it query routing. And he compares it to managing a corporate fleet of vehicles.
SPEAKER_01Yes, this is a great way to picture it.
SPEAKER_00Right. Like if your employee needs to run a quick errand three blocks down the street, you give them the keys to a scooter, you save the massive expensive semi-truck for the heavy-duty cross-country freight jobs.
SPEAKER_01Let's actually look at how a query router functions mechanically to make that happen because it's brilliant.
SPEAKER_00Yeah, walk us through it.
SPEAKER_01When a user in your company types a prompt into your internal system, that prompt doesn't just go straight to an expensive model. It first heads a lightweight classifier model.
SPEAKER_00Like a traffic cop.
SPEAKER_01Exactly, a traffic cop. It analyzes the text and determines the complexity. It might say, oh, this user just wants to extract a name from a digital receipt. Route this to our piny cheap model.
SPEAKER_00The scooter.
SPEAKER_01The scooter, right. But for the next prompt, it might recognize, wow, this requires analyzing a 50-page financial disclosure. Route this to the massive high-powered model.
SPEAKER_00To semi truck.
SPEAKER_01You got it.
SPEAKER_00So that totally prevents you from wasting massive amounts of compute and money on simple tasks. But you know, beyond just saving money on compute, Holman brings up a regulatory ticking clock that should honestly be flashing red on every executive's dashboard right now.
SPEAKER_01Oh, absolutely. Data privacy and the EU AI Act.
SPEAKER_00Yes. We have to look at the timeline here. August 2, 2026. Put that date on your strategic roadmap right now.
SPEAKER_01It's coming fast.
SPEAKER_00It is. That's when the EU AI Act becomes fully effective, and it demands unprecedented levels of transparency, accountability, and bias mitigation from enterprise AI systems.
SPEAKER_01And noncompliance means severe penalties, like business ending penalties.
SPEAKER_00So how does relying on like public APIs put an enterprise at risk here?
SPEAKER_01Well, think about it. When you use a public API from one of the major providers, you're basically sending your proprietary data out into a black box hosted on someone else's servers. Right? It leaves the building. Exactly. You do not have total control over how that data is processed, how it might be stored, or how the model's inherent biases might affect a total nightmare. For high-risk operations like HR promotion decisions or handling customer financial data, that lack of transparency is a massive regulatory risk under the new EU framework.
SPEAKER_00So a multi-LLM mesh actually solves this by allowing you to mix and match private and public infrastructure.
SPEAKER_01Right. An LLM mesh gives you the architectural flexibility to utilize self-hosted private models for your high-risk sensitive operations. You keep complete control over that data.
SPEAKER_00It stays in-house.
SPEAKER_01It never leaves your physical or virtual servers, which totally satisfies the regulators. But simultaneously, the mesh allows you to route the low-risk, non-sensitive tasks like drafting a generic marketing email out to those cheap public APIs.
SPEAKER_00You're tailoring your architecture to meet strict regulatory needs without forcing a single model to do everything securely, which is honestly nearly impossible.
SPEAKER_01It really is impossible.
SPEAKER_00So taking a step back here, the LLM mesh solves the performance plateau through specialization. It solves the operational cost issue through smart query routing using the scooter instead of the semi-truck. And it solves the compliance nightmare of the EU AI Act by keeping sensitive data on self-hosted models. Sounds perfect, right? Architecturally, it sounds like a silver bullet. But my CFO alarm bells are ringing so loud right now.
SPEAKER_01I was waiting for that.
SPEAKER_00Because connecting a dozen different self-hosted and public models together sounds like an absolute budget killer to actually build.
SPEAKER_01Yeah.
SPEAKER_00What are the hidden costs of this mesh?
SPEAKER_01You're totally right to be skeptical. Holman is very clear that an LLM mesh is not some simple plug-and-play solution. The costs are incredibly steep and they come in three distinct waves.
SPEAKER_00Okay, what's wave one?
SPEAKER_01The first wave is development. Designing this takes serious, really expensive engineering resources.
SPEAKER_00Because orchestration isn't just like generating a few API keys and calling it a day.
SPEAKER_01Not at all. Orchestration means making all these distinct models communicate seamlessly. You need to build pomped translation layers so that different models actually understand each other.
SPEAKER_00Oh wow. I didn't even think about that.
SPEAKER_01Yeah. And you have to maintain the context of a conversation as a user's query bounces between a cheap model and an expensive model. Plus, you have to build robust security guardrails across the whole thing.
SPEAKER_00That sounds incredibly complex.
SPEAKER_01It is. Holman notes that even if you leverage enterprise platforms like the Taekw to streamline this orchestration, the upfront investment in engineering talent and software licenses is incredibly heavy.
SPEAKER_00Which leads us to the second wave: operating costs. Let's break down the math of how these API fees actually work because companies seem to get completely blindsided by this.
SPEAKER_01They really do. So every single time your system pings an external provider, you are charged a fraction of a cent based on the number of tokens or words processed.
SPEAKER_00Which seems microscopic on paper.
SPEAKER_01Right. On paper, it's nothing. But if you roll out an internal AI tool to 10,000 employees and they each run 50 queries a day, suddenly those fractions of a cent balloon into thousands of dollars in monthly operational expenses.
SPEAKER_00And that doesn't even account for the massive cloud infrastructure cost of running the private self-hosted models in your mesh.
SPEAKER_01Exactly. You're paying on both ends.
SPEAKER_00So what's the third wave?
SPEAKER_01The third wave is training and maintenance. Because these models experience what engineers call data drift or model decay. Right.
SPEAKER_00They get outdated.
SPEAKER_01Yeah. The world changes, your internal business jargon changes, industry regulations change. You cannot just build it once and walk away. You need a dedicated team continuously tweaking, updating, and fine-tuning the models to keep them accurate.
SPEAKER_00Aaron Powell It's a continuous perpetual investment.
SPEAKER_01Yeah. And the maintenance cost often dwarfs the initial development cost over time.
SPEAKER_00Aaron Powell Okay, let me step into the shoes of a really skeptical, mid-sized enterprise leader for a second.
SPEAKER_01Go for it.
SPEAKER_00I don't have massive enterprise scale query volumes. I certainly don't have a Google-sized IT budget. This mesh sounds like a governance nightmare. If I try to build this, the sheer complexity is going to eat my margins alive. What is my alternative?
SPEAKER_01Holman totally acknowledges that reality. For organizations where the math of a massive mesh just doesn't add up, he outlines two highly strategic alternatives. The first is to prioritize fewer, more versatile models.
SPEAKER_00Meaning instead of spreading my limited engineering resources thin across a dozen specialized models.
SPEAKER_01I just consolidate. Exactly. You pick one or two highly capable foundational models and you invest all your energy into fine-tuning them deeply for your specific business workflows. Okay. You intentionally sacrifice a bit of that perfect granular task specialization in exchange for operational simplicity, easier governance, and drastically lower overhead.
SPEAKER_00That makes a lot of sense. And what is the second alternative Holman offers?
SPEAKER_01Adopt open source foundational models. Holman points to platforms like Hugging Face and highly capable open source models like Meta's Llama or TII's Falcon.
SPEAKER_00Right, the open source movement is huge right now.
SPEAKER_01It is. By utilizing these, you bypass proprietary API fees entirely. And critically, for compliance, you get complete ownership of the stack. You self-host, ensuring total data privacy.
SPEAKER_00Wait, let me pump the brakes on that for a second. Uh-oh, here it comes. Well, open source sounds fantastic on a marketing brochure for avoiding API fees, but aren't we just trading one massive cost for another?
SPEAKER_01What do you mean?
SPEAKER_00It's like the old saying it's free like a free puppy, not free like a free beer.
SPEAKER_01Oh, right.
SPEAKER_00You still have to feed it and take it to the vet. If I drop API fees, now I have to hire a team of $250,000 a year machine learning engineers just to keep a llama instance from breaking. I have to manage the hardware. How is that actually cheaper for me in the long run?
SPEAKER_01It's a crucial trade-off, and you are exactly right to question the true cost. You are essentially trading variable API fees for fixed engineering salaries and server costs.
SPEAKER_00Which isn't a small trait.
SPEAKER_01Not at all. For a small startup with low query volume, open source is absolutely not cheaper. But for an enterprise with high continuous query volumes and strict data privacy requirements, the break-even point arrives surprisingly fast.
SPEAKER_00Oh, okay. That makes sense.
SPEAKER_01Having complete ownership of the stack, customizing it to your exact security specs, and avoiding vendor lock-in where an API provider can just suddenly raise their prices by 20% for many enterprises, that level of control makes the engineering investment incredibly worthwhile.
SPEAKER_00So it's about control as much as cost.
SPEAKER_01Exactly.
SPEAKER_00Right. As we bring this comprehensive briefing to a close, there is one critical caveat from Victor Holman that we need to emphasize. If you open any tech publication right now, you will undoubtedly see massive headlines screaming about an AI slowdown or predicting a new AI winter.
SPEAKER_01Yeah, and they're writing those headlines because the massive general models aren't leaping forward the way they were 18 months ago. The brute force scaling has plateaued.
SPEAKER_00Holman is warning us: do not let that media narrative fool you into complacency. Progress is not stopping. What is actually happening?
SPEAKER_01The progress is simply changing direction. This plateau we are hitting, it is actually a launching pad.
SPEAKER_00I love that framing.
SPEAKER_01Yeah. The innovation is moving away from raw scaling, which is really just trying to build a bigger brain during the training phase, and it's moving toward new inference-based strategies.
SPEAKER_00Break that down for us a bit. What is the difference between training and inferencing in this context?
SPEAKER_01Think of it like a student. Training is reading the textbooks and memorizing facts. That phase of AI development is what is stalling.
SPEAKER_00Okay.
SPEAKER_01Inference is taking the open book exam. It happens after the model is trained when you actually ask it a question.
SPEAKER_00Oh, got it.
SPEAKER_01So inference-based strategies focus on improving how the model reasons through a can klex problem in real time, how it searches the web for external information, and how it interacts with other software tools as an autonomous agent.
SPEAKER_00So it's about what it can do, not just what it knows.
SPEAKER_01Exactly. That is where the next massive leap in enterprise value is going to come from improving the application of the intelligence rather than just brute forcing more raw intelligence.
SPEAKER_00So here is the call to action for you. As you assess your own organization's roadmap based on Holman's insights today, it is time to seriously reassess your AI strategy. Are you still chasing the hype of the biggest foundational model?
SPEAKER_01You really have to ask yourself the hard questions.
SPEAKER_00Right. Do you truly have the diverse high-stakes use cases that justify the massive complexity, engineering costs, and governance of a full LLM mesh? Or you know, could you achieve your business goals faster and significantly cheaper with fewer finely tuned models?
SPEAKER_01Or perhaps by taking control of your own infrastructure with open source options like Llama.
SPEAKER_00Exactly. You have to strip away the hype and do the rigorous mathematical and regulatory analysis on your specific operational reality.
SPEAKER_01Because the old playbook is dead.
SPEAKER_00It is. And as you do that math, we want to leave you with one final provocative thought to mull over. We talked earlier about how these underlying foundational AI models are inevitably becoming commoditized utilities, right? They're becoming identical brands of bottled water or like the electricity powering your office building.
SPEAKER_01Aaron Powell The intelligence itself is becoming a baseline commodity.
SPEAKER_00So if the AI is just the raw electricity powering the building, then the competitive advantage of the future will not belong to the company that buys the most expensive electricity.
SPEAKER_01No, it won't.
SPEAKER_00It will belong entirely to the company with the most proprietary, unique data to feed into the system. And more importantly, it will belong to the company that designs the most effective human-agent workflows to plug into that socket.
SPEAKER_01It reframes the entire AI race. If the models are essentially equal across the board, your unique organizational data and your proprietary operational workflows are your only true defensible moats.
SPEAKER_00Where does that leave your current AI investments? Are you wasting your budget buying expensive electricity or are you actually building the factory of the future?