Strategy in the Face of Chaos
Audio companions to my writing on strategy, technology, AI, cybersecurity and building technology businesses.
Each edition explores one of my published articles through an AI-generated discussion or debate, offering another way to engage with its central ideas. These are not interviews or original podcast episodes, and the voices are not mine. The written article remains the definitive version.
This channel is currently a pilot, and the format will evolve as I learn what works.
Strategy in the Face of Chaos
AI Power Demands: Chiplets and the Future Beyond Moore’s Law
Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.
This pilot audio edition explores the argument behind AI Power Demands: Chiplets & the Future Beyond Moore’s Law.
The discussion examines how AI’s rapidly growing demand for computing power and energy is colliding with the physical and economic limits of traditional transistor scaling. GPUs have sustained much of AI’s progress, but each generation brings greater power consumption, heat and manufacturing complexity. Performance can no longer depend on Moore’s Law alone.
It explores chiplets and heterogeneous integration as an alternative path. By combining smaller components designed for different functions, manufacturers can improve flexibility, production yields and cost efficiency. Advanced packaging, faster interconnects and common standards could make this modular approach fundamental to the future of AI and high-performance computing.
The discussion also compares NVIDIA’s cautious adoption of chiplets with AMD’s longer commitment to modular design. AMD may hold structural advantages in flexibility and manufacturing efficiency, while NVIDIA retains a powerful position through CUDA, networking and its integrated computing platform.
This is an AI-generated discussion based on my published article. It offers another way to engage with the ideas, but it is not an interview or a recording of me. The written article remains the definitive version.
By 2030, you know, the artificial intelligence running your company's software could be consuming like 16% of the entire United States power grid.
SPEAKER_01Aaron Powell Yeah, that is just uh it's a staggering number to try and wrap your head around right off the bat.
SPEAKER_00Aaron Powell It really is. And welcome to the deep dive, everyone. We've designed today's discussion specifically for you, the technology and business leaders who are trying to navigate one of the most volatile and frankly physically constrained landscapes in modern computing.
SPEAKER_01Aaron Powell Right. Because when we talk about enterprise AI strategies, I mean the the conversation almost always defaults to the cloud.
SPEAKER_00Aaron Powell Exactly. We treat AI like we're building this uh this limitless invisible skyscraper. You know, we obsess over the height, the design, the parameters of the latest large language model.
SPEAKER_01Aaron Powell It's so easy to fall into that trap, too, because the software feels completely virtual. You pay a subscription, you hit an API, and boom, magic happens on your screen.
SPEAKER_00Aaron Powell But today we are going deep into the basement. Because underneath that invisible software skyscraper is a very real, very physical, and incredibly hot foundation of silicon.
SPEAKER_01And right now, that physical foundation is showing some severe structural cracks.
SPEAKER_00Yeah, it really is. We are basing today's executive briefing entirely on a foundational analysis by Victor Holman. It's titled AI Power Demands Chiplets and the Future Beyond Moore's Law.
SPEAKER_01It's a great piece, very grounded.
SPEAKER_00Right. So we aren't here today to marvel at what AI can write or draw. We are here to unpack a hidden physical bottleneck that literally threatens to derail your IT budgets.
SPEAKER_01Which is something every executive needs to be looking at. If you are projecting your company's compute capabilities, say three to five years out, you can't just look at software roadmaps anymore. Trevor Burrus, Jr.
SPEAKER_00Right. You have to understand the fundamental physics of the hardware powering it. The physical reality of AI is, well, it's violently colliding with our software ambitions.
SPEAKER_01Aaron Powell Absolutely. And let's start with the sheer scale of the energy crisis Holman outlines, because he pulls this staggering estimate from Wells Fargo.
SPEAKER_00Oh, right, the 550% number?
SPEAKER_01Yes. AI-related power consumption is projected to grow by 550% by 2026. Trevor Burrus, Jr.
SPEAKER_00Which is just around the corner.
SPEAKER_01Literally tomorrow in enterprise planning terms, and it's not an outlier prediction either. Goldman Sachs and the International Energy Agency are forecasting the exact same exponential growth.
SPEAKER_00Trevor Burrus, Jr.: 16% of national electricity usage for a single technological sector. I mean, that should make any executive pause.
SPEAKER_01Oh, for sure. Historically, data centers have been, you know, relatively predictable when it came to power and cooling. You could model out your capital expenditures with a fair degree of certainty.
SPEAKER_00But generative AI just threw a wrench in all of that.
SPEAKER_01It completely blew up those models. Yeah.
SPEAKER_00So to understand the mechanics of why this is happening, we really have to look at the death of an industry golden rule, which is Moore's Law.
SPEAKER_01Right, the old reliable heartbeat of tech.
SPEAKER_00Exactly. Back in 1965, Gordon Moore predicted that the number of transistors on a chip would double roughly every two years.
SPEAKER_01And for decades that held true. You didn't even have to change your software. You just waited two years, bought new hardware, and your programs magically ran faster, smaller, and cheaper.
SPEAKER_00But then, in 2017, NVIDIA CEO Jensen Huang famously stood up in his signature leather jacket and just flat out declared, more era's law is dead.
SPEAKER_01It definitely ruffled some feathers at the time. But uh Holman's analysis proves that the semiconductor industry has basically slammed into three distinct, insurmountable walls.
SPEAKER_00The era of getting free performance gains just by shrinking transistors down is over.
SPEAKER_01Right. You can't just make things smaller and expect it to work anymore.
SPEAKER_00And I really want to dig into the why here for our listeners, because it's not a lack of engineering talent. We aren't out of good ideas.
SPEAKER_01Aaron Powell No, not at all. We are literally fighting the laws of physics and the boundaries of basic economics.
SPEAKER_00Aaron Powell So let's look at that first wall. Physics. We're talking about quantum effects.
SPEAKER_01Yeah. When you shrink a transistor down to the atomic scale, classical physics just stops applying. You run into what's called quantum tunneling.
SPEAKER_00Aaron Powell Wait, what does that actually mean in a microchip?
SPEAKER_01Well, at those microscopic scales, the physical barriers holding the electrons back are so thin that the electrons just become unpredictable.
SPEAKER_00Like they don't stay where they're supposed to.
SPEAKER_01Exactly. They literally pass right through solid matter that they shouldn't be able to cross. So you pump energy into the chip to perform a calculation, and massive amounts of that power just leak out.
SPEAKER_00It's like trying to pipe high pressure water through a garden hose made out of wet paper towels. I mean, no matter how much water you pump in, it's just gonna bleed out the sides.
SPEAKER_01Aaron Powell That is a perfect analogy. And that bleeding leads directly to the second wall, which is thermal density. Trevor Burrus, Jr.
SPEAKER_00Because all that leaking energy turns into heat, right?
SPEAKER_01Exactly. When you pack billions of these leaking microscopic transistors closer and closer together, the heat generated in that tiny surface area becomes absolutely unmanageable.
SPEAKER_00Aaron Powell So what happens if you can't cool it?
SPEAKER_01If you cannot extract that heat from the silicon, the processor will either throttle its own performance to survive or it will physically melt.
SPEAKER_00Wow. Okay, which brings us to the third wall, the economics.
SPEAKER_01Aaron Powell Right, because getting down to these atomic scales requires extreme ultraviolet lithography or EUV.
SPEAKER_00And Holman mentions these machines are incredibly expensive.
SPEAKER_01They are some of the most complex devices ever built by humans. And they are so astronomically expensive to run that the actual cost per transistor is no longer dropping.
SPEAKER_00So it's not just physics saying stop, it's the accountants too.
SPEAKER_01Basically, yeah. We have reached a point where the fundamental laws of physics and the boundaries of capital are just saying stop. You cannot reliably or cheaply make the engine any smaller.
SPEAKER_00Aaron Powell But the market is screaming for more horsepower to run these massive language models. So if you can't make the engine smaller and more efficient, what do you do?
SPEAKER_01The only option left is brute force. You just build a much, much bigger engine.
SPEAKER_00Aaron Powell Which introduces us to the reign of the GPUs.
SPEAKER_01Graphics processing units, yeah. They were originally designed to render video games, you know, pushing pixels to a screen, but their underlying architecture happened to be the perfect brute force mechanism for machine learning.
SPEAKER_00Aaron Powell Let's explain the mechanism there just briefly. Why are GPUs the kings of AI compute right now? While a traditional CPU like the brain in most of our laptops, why does it struggle with these workloads?
SPEAKER_01It comes down to sequential versus parallel processing. Think of a CPU as like a single world-class mathematics professor. Okay. Give them an incredibly complex multi-step calculus problem, and they will solve it faster than anyone. But AI doesn't really rely on a few complex problems.
SPEAKER_00It relies on billions of very simple mathematical operations happening all at once.
SPEAKER_01Exactly. So a GPU isn't a single professor, it's an army of 10,000 middle school math students.
SPEAKER_00I love that.
SPEAKER_01Right. They aren't doing calculus, but if you give them 10,000 basic addition problems, the army finishes in a fraction of a second.
SPEAKER_00And Holman points out that modern AI GPUs have specialized hardware called tensor cores built directly into the silicon to manage that army.
SPEAKER_01Yeah, they are hyper-optimized for the exact type of matrix math used in machine learning.
SPEAKER_00Which is crucial for both training the models and for inference, right?
SPEAKER_01Right. Right. Inference being when the model is actually running live and generating a response for an end user. But here is the catch with the GPU approach.
SPEAKER_00The power draw.
SPEAKER_01Huge power draw. Commanding an army of 10,000 simultaneous pathways requires an unfathomable amount of energy. When you light up an entire city grid of microscopic circuits at the exact same millisecond, the current draw is just violent.
SPEAKER_00And that brings us to a sequence of stats from the article that honestly should terrify any IT director trying to budget for cooling. Let's look at NVIDIA's generational power draw.
SPEAKER_01This really puts it in perspective.
SPEAKER_00Yeah. So a few years ago, their flagship A100 GPU consumed about 400 watts of power at peak.
SPEAKER_01Which is heavy for a single component, but you know, totally manageable in a standard server rack.
SPEAKER_00Right. But then came the H-100, the chip that basically launched the generative AI boom, that jumped to 700 watts.
SPEAKER_01Almost double.
SPEAKER_00And now look at the new Blackwell B200 architecture. A single B200 pulls up to 1200 watts. It's wild. It gets worse. If you look at their G B200 configuration, which pairs two B200 GPUs with a central CPU, it hits a staggering 2700 watts.
SPEAKER_01We are looking at a 300% increase in power consumption in basically a single generational leap. It's unheard of.
SPEAKER_00To put that in perspective, for you listening, that is like taking a standard, you know, 100-watt light bulb out of your server rack and replacing it with a commercial space heater. Yep. Now imagine a data center filled with thousands of those space heaters just stacked on top of each other. How on earth do data centers plan to manage that? How do you even supply enough electricity to the building?
SPEAKER_01Aaron Powell Honestly, you practically have to build a dedicated power plant next door.
SPEAKER_00Aaron Ross Powell Wait, really?
SPEAKER_01Yeah. Companies are actively exploring nuclear micro reactors for this exact reason. Sheer power scaling, you know, the brute force approach of just pumping more wattage into these massive pieces of silicon, it's completely unsustainable.
SPEAKER_00Aaron Powell So the physical architecture of the hardware just has to change.
SPEAKER_01It has to evolve, yeah.
SPEAKER_00Okay, before we talk about what that evolution looks like, let's define a term that comes up a lot in the article the die.
SPEAKER_01Ah.
SPEAKER_00When engineers talk about a monolithic die, they just mean the actual physical rectangle of silicon that gets cut from the manufacturing wafer.
SPEAKER_01Correct. A monolithic chip means all the components, uh the compute cores, the memory controllers, the impute output channels, they are all etched onto one single unbroken piece of silicon.
SPEAKER_00Aaron Ross Powell And because of those thermal and economic walls we just mentioned, creating these giant, power-hungry, monolithic dyes is hitting a wall.
SPEAKER_01Aaron Powell Exactly. So Holman outlines the semiconductor industry's massive pivot towards something called heterogeneous integration.
SPEAKER_00Aaron Powell or in plain English. Chiplets. They are the pivotal escape route for the entire industry right now. Instead of trying to manufacture one massive, flawless, incredibly hot monolithic processor, you break it apart.
SPEAKER_01You break the processor down into smaller specialized modules, the chiplets.
SPEAKER_00Right. You then stitch them together on a single package so they act as one unified brain.
SPEAKER_01I've heard modular design compared to like building blocks or Legos. But given the complexity here, I think a commercial kitchen is a much better analogy.
SPEAKER_00Well, like that. How so? Well, a monolithic chip is like a tiny restaurant where one single master chef is trying to do everything. They chop the vegetables, boil the pasta, sear the steak, and bake the dessert all at once. Aaron Powell Right.
SPEAKER_01And as the restaurant gets busier, that chef eventually collapses from exhaustion.
SPEAKER_00Or in our case, thermal throttling.
SPEAKER_01Yeah.
SPEAKER_00But the chiplet approach is like opening a massive commercial kitchen and hiring specialized staff.
SPEAKER_01Okay. I see where you're going.
SPEAKER_00You have an executive chef handling the most complex dishes, a sous chef prepping vegetables, a pastry chef handling the baking. They all work at their own stations, but they communicate so rapidly that the food comes out perfectly timed.
SPEAKER_01That is a great way to picture it. And that commercial kitchen model gives us the three distinct advantages Holman highlights in his article.
SPEAKER_00Let's walk through those. The first one is modular design, right?
SPEAKER_01Right. In semiconductor terms, not every part of the processor needs to be manufactured on the absolute most expensive, cutting-edge, like three nanometer process. Yeah. Your high performance compute cores, your executive check, they need that expensive three nanometer silicon. But the parts of the chip that just manage input and output or basic memory routing.
SPEAKER_00They don't need to be geniuses.
SPEAKER_01Exactly. You can manufacture those specific chiplets on older, cheaper, proven seven nanometer technology.
SPEAKER_00So you mix and match to save a ton of money. And that leads to the second advantage: yield. Because manufacturing silicon is uh it's a messy process, right? Microscopic defects happen all the time.
SPEAKER_01All the time. And if you are making a giant monolithic die and one single transistor fails out of 80 billion.
SPEAKER_00Just you have to throw the whole thing away.
SPEAKER_01You literally throw the entire multi-thousand dollar piece of silicon in the trash, the yield is terrible. Wow. But with chiplets, if the memory controller module has a defect, you just throw away that one cheap little module.
SPEAKER_00Oh, and you keep the expensive compute chiplets.
SPEAKER_01Exactly. You drastically reduce silicone waste, which completely transforms the unit economics of AI hardware.
SPEAKER_00Aaron Powell That makes total sense. And the third advantage is performance scaling.
SPEAKER_01Aaron Powell Right. As enterprise workloads change, you can literally swap out stations in your kitchen.
SPEAKER_00Aaron Powell So if you need more memory bandwidth, you just add a specialized memory chiplet.
SPEAKER_01Yeah, you don't have to redesign an entirely new monolithic processor from scratch just to handle a new type of AI model.
SPEAKER_00Aaron Powell But breaking the chip apart has to introduce new problems, right? Yeah. I mean, the chefs in our kitchen need to talk to each other instantly.
SPEAKER_01Aaron Powell That is the massive catch. If data has to travel across physical space between different chiplets, it creates latency. And in AI training, a microsecond of latency is an eternity.
SPEAKER_00Aaron Powell So how do they connect them fast enough to fake being one single monolithic brain?
SPEAKER_01Aaron Powell Through advanced packaging. Holman details two main approaches. The first is 2.5 D packaging.
SPEAKER_002.5D.
SPEAKER_01Yeah. Imagine a microscopic, ultra fast highway system built underneath the chiplets. This is called a silicon interposer. You place the chiplet side by side on this interposer, which allows massive lanes of data to flow between them instantly.
SPEAKER_00Oh, so this is how manufacturers integrate high bandwidth memory directly adjacent to the compute cores.
SPEAKER_01Exactly, side by side on a data highway.
SPEAKER_00And the other method?
SPEAKER_013D stacking. Instead of placing them side by side, you literally stack the silicon chiplets vertically on top of each other.
SPEAKER_00Well, like a high rise.
SPEAKER_01Right. This drops the physical distance data has to travel to almost zero. AMD has actually pioneered this with their 3D Vcash technology, stacking memory right on top of the processor.
SPEAKER_00And Holman says that gave them like a 15% performance boost in data center workloads, right?
SPEAKER_01Yep. Double-digit boost just from vertical stacking.
SPEAKER_00Wait, let me look at this from an engineering standpoint for a second. We just spent 10 minutes establishing that densely packed silicon turns into a commercial space heater.
SPEAKER_01Yeah.
SPEAKER_00If we take these already blistering hot components and now we literally sandwich them vertically on top of each other, doesn't that trap the heat and make the thermal wall exponentially worse?
SPEAKER_01Oh, it absolutely does. It is the brilliant paradox of this era, and Holman doesn't sugarcoat it at all.
SPEAKER_00So we're just building a denser space heater.
SPEAKER_01Essentially, yes. Thermal management is the ultimate engineering bottleneck of the chiplet paradigm. You solve the economic yields and you bypass the physical limits of a single die, but you inherit an absolute nightmare of a thermal puzzle.
SPEAKER_00So how do they deal with it?
SPEAKER_01Making 3D stacking work at an enterprise scale requires exotic liquid cooling loops and these die-to-die connections that don't generate their own parasitic heat. It is a massive trade-off.
SPEAKER_00Okay, so we know what chiplets are and we know why they are physically necessary. Let's look at the strategic battleground.
SPEAKER_01This is where it gets really interesting for business leaders.
SPEAKER_00Right. How is this exact technology defining the war between the major hardware providers your IT departments are buying from?
SPEAKER_01Well, just to show this is an industry-wide shift, we should briefly mention Intel.
SPEAKER_00Yeah, they're Ponto Vecchio GPU. Holman notes it crams 100 billion transistors into a single module using 47 distinct chiplets.
SPEAKER_0147? It's incredible.
SPEAKER_00But the real tension in Holman's piece is the dramatic divergence in strategy between AMD and NVIDIA.
SPEAKER_01They are taking wildly different approaches to this physical bottleneck. Let's look at AMD first. They placed their strategic bet on chiplets more than a decade ago. Right.
SPEAKER_00Take their flagship MI300X AI accelerator.
SPEAKER_01Yeah. It is a fully realized vision of that commercial kitchen. It uses a deeply modular 12-chiplet design. They have distinct chiplets dedicated entirely to routing data, and separate chiplets dedicated purely to raw compute.
SPEAKER_00Because they've spent 10 years refining how to stitch those pieces together.
SPEAKER_01Exactly. And the business implication is profound. Because AMD is working with smaller individual pieces of silicon, their manufacturing yields are vastly superior.
SPEAKER_00So the production costs are lower.
SPEAKER_01Much lower. And their ability to customize a layout for a specific enterprise client is highly flexible.
SPEAKER_00But then we look at the Reigning King Nvidia. The article points out that their new Blackwell architecture, you know, the one drawing 2700 watts, is actually a very hesitant, tentative shift toward modularity.
SPEAKER_01Yeah. The P200 GPU isn't 12 chiplets like AMD. It's just two massive chiplets stitched together.
SPEAKER_00Aaron Powell So instead of opening a modular commercial kitchen, NVIDIA basically just hired two giant chefs and chained them together.
SPEAKER_01Aaron Powell That's yeah, that's exactly what they did. They are clinging to large monolithic designs to maximize raw, unadulterated speed. Why?
SPEAKER_00Just for the benchmarks.
SPEAKER_01Because every time data jumps between chiplets, you risk latency. They are terrified of latency. They want the data to stay on the same piece of silicon as long as physically possible.
SPEAKER_00Okay, so here is the multi-billion dollar question for anyone planning a long-term IT strategy right now.
SPEAKER_01Let's hear it.
SPEAKER_00If AMD has a clear structural hardware advantage, I mean, if their chips are cheaper to make, have less manufacturing waste, and are more flexible, why is NVIDIA still commanding roughly 80% or more of the enterprise AI market?
SPEAKER_01Aaron Powell Because the hardware is only half the product. NVIDIA's protective moat is deeply entrenched in software and networking.
SPEAKER_00Aaron Powell Specifically, their C UDA platform.
SPEAKER_01Right. C D is a proprietary software ecosystem that AI developers have been relying on for years. It's essentially the native language of AI development at this point.
SPEAKER_00Aaron Powell So when your IT department buys an NVIDIA cluster, they aren't just buying silicon.
SPEAKER_01No, they're buying a perfectly vertically integrated ecosystem. The software, the compute hardware, and the networking, like their architectures that fuse CPUs and GPUs together. They're all perfectly tuned to move data flawlessly.
SPEAKER_00Aaron Powell Because in AI, preventing traffic jams on the network is just as critical as doing the math itself.
SPEAKER_01Exactly. It's Bender lock-in at its absolute finest. If your developers only speak C UDA, you buy NVIDIA, even if the hardware runs hotter.
SPEAKER_00It's AMP knows this, right?
SPEAKER_01Oh, they definitely know. They are heavily pushing an open source alternative called ROCM to try and break that software lock-in.
SPEAKER_00And Holman mentions they acquired a company called Pensando, right?
SPEAKER_01Yes. Specifically to build out their data center and networking capabilities to rival NVIDIA. But for now at least, NVIDIA's integrated ecosystem is the shield blocking AMDS modular hardware advantage. Got it.
SPEAKER_00Okay. Let's pull this all together into an executive summary for our listeners. The 550% surge in AI energy demand isn't just an anomaly.
SPEAKER_01Not at all.
SPEAKER_00It is a permanent shift that is forcing a fundamental hardware evolution. The era of more you know relying on smaller transistors for free gains is completely dead.
SPEAKER_01We have firmly entered the more than more era.
SPEAKER_00And the sheer brute force scaling of monolithic GPUs is thermally and economically unsustainable. Heterogeneous integration breaking the chip apart into chiplets is the only viable physical path forward for high performance computing.
SPEAKER_01So for business leaders listening to this, the takeaway is pretty clear. The long-term cost of your enterprise AI won't just be dictated by who designs the smartest software model. Right. It will be dictated by which hardware vendor actually masters advanced packaging and liquid cooling.
SPEAKER_00Absolutely. Now, before we sign off, I want to leave you with a final thought to ponder. Based on a standard Holman mentions in his article called UCI.
SPEAKER_01Ah, the Universal Chiplet Interconnect Express.
SPEAKER_00Exactly. It is an emerging industry standard designed to make sure chiplets from different manufacturers can actually talk to each other on the same interposer highway.
SPEAKER_01It's like a universal translation layer for the silicon itself.
SPEAKER_00Right. So think about the strategic implications if UCI truly becomes the global standard. Could the future of enterprise AI data centers look less like buying a massive proprietary locked-in system from a single vendor, and more like building a custom PC?
SPEAKER_01That is a fascinating thought.
SPEAKER_00Imagine a near future where your IT architects can seamlessly mix, say, an AMD compute chiplet, an NVIDIA AI accelerator, and an Intel memory controller, all snapping together on the exact same silicon board.
SPEAKER_01Perfectly tailored for your company's proprietary data workload. It would completely shatter the vendor lock-in that defines the market today.
SPEAKER_00Exactly. It democratizes the hardware layer.
SPEAKER_01That is a provocative possibility. And it should entirely reframe how companies approach their hardware contract negotiations over the next five years.
SPEAKER_00It really changes everything. You know, we started this deep dive by picturing an AI skyscraper built on a cracking foundation, the soaring energy demands of the death of Moore's Law, those are the cracks. But chiplets, chiplets are the industry pouring a highly engineered modular new foundation to keep the whole thing from collapsing.
SPEAKER_01Perfectly said.
SPEAKER_00Well, thank you for joining us on this deep dive into the physical reality of AI. As you head into your next strategic planning cycle, look past the software hype and really consider how your organization is preparing for the hardware bottlenecks coming down the line. We'll see you next time.