Strategy in the Face of Chaos
Audio companions to my writing on strategy, technology, AI, cybersecurity and building technology businesses.
Each edition explores one of my published articles through an AI-generated discussion or debate, offering another way to engage with its central ideas. These are not interviews or original podcast episodes, and the voices are not mine. The written article remains the definitive version.
This channel is currently a pilot, and the format will evolve as I learn what works.
Strategy in the Face of Chaos
The Challenges of Calculating ROI for AI Initiatives
Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.
This pilot audio edition explores the argument behind The Emerging Threat of AI-Powered Cybercrime.
The discussion examines how state-affiliated groups linked to China, Iran, North Korea and Russia have used AI to support cyberattacks and influence operations. These uses include reconnaissance, vulnerability research, scripting, malware debugging, phishing and the creation of deceptive online content.
The evidence suggests that AI is currently strengthening familiar stages of cyber operations rather than creating fundamentally new attack techniques. However, greater speed, efficiency and accessibility could lower the barriers to sophisticated attacks. More capable systems could also enable persuasive influence campaigns and deeper attacks against critical infrastructure.
The discussion considers how organisations and governments should respond through AI-specific security, adversarial testing, collaborative intelligence sharing, trustworthy models, responsible governance and greater transparency around AI misuse.
This is an AI-generated discussion based on my published article. It offers another way to engage with the ideas, but it is not an interview or a recording of me. The written article remains the definitive version.
So McKinsey predicts, uh they're actually saying artificial intelligence will inject something like $13 trillion into the global economy by the end of this decade.
SPEAKER_01Aaron Powell Right. Which is just a massive number.
SPEAKER_00Aaron Powell It's insane. And then you have Goldman Sachs projecting that AI could automate 25 percent of all current work tasks. They're talking about raising U.S. productivity by 9 percent.
SPEAKER_01Aaron Powell Yeah, and driving cumulative GDP growth of over 6 percent in the next 10 years. It's huge.
SPEAKER_00Aaron Powell Exactly. I mean the macroeconomic environment is basically screaming that this is the most lucrative technological shift in a generation.
SPEAKER_01Aaron Powell Oh, absolutely.
SPEAKER_00Aaron Powell But and here's the really interesting part. If you walk into almost any corporate boardroom right now, the CFO and the CTO are staring at their AI budgets in a state of absolute paralysis.
SPEAKER_01Aaron Powell It is such a fascinating disconnect, really, because you know you have these staggering macro level forecasts promising all this unprecedented wealth generation.
SPEAKER_00Yeah.
SPEAKER_01But on the micro level, like, inside the actual organizations that are expected to build this future leadership teams are completely frozen.
SPEAKER_00They just don't know what to do.
SPEAKER_01Right. I mean, they run a successful pilot, everyone applauds the proof of concept, and then the initiative just kind of dies on the vine because nobody can figure out how to financially justify scaling it up.
SPEAKER_00Welcome to today's deep dive. If you are a technology or business leader listening to this, you know exactly what we are talking about.
SPEAKER_01Definitely.
SPEAKER_00Our mission today is to really crack the code on what might be the single most frustrating roadblock in corporate strategy right now.
SPEAKER_01The ROI problem.
SPEAKER_00Exactly. How to accurately calculate the return on investment for artificial intelligence. And we have an incredibly rigorous source guiding our conversation today.
SPEAKER_01Yeah, it's a great piece.
SPEAKER_00It's an article by Victor Holman titled The Challenges of Calculating ROI for AI Initiatives. And I want to set the tone immediately for you listening. We are not here to marvel at the science fiction of what AI can do.
SPEAKER_01Right. No flying cars today.
SPEAKER_00No flying cars. We are here to rigorously examine how you pay for it, how you justify it to a skeptical board of directors, and you know, how you measure its actual financial impact without falling into some very expensive traps.
SPEAKER_01And Holm Aaron's central thesis zeroes right in on the root cause of that boardroom paralysis you mentioned. Right. He argues that companies are stuck in this limbo because they are attempting to measure a completely novel asset class using, well, outdated conventional accounting and financial management practices. Trevor Burrus, Jr.
SPEAKER_00It's like using a ruler to measure the temperature.
SPEAKER_01Exactly. Traditional ROI math simply fails to capture AI's unique long-term value creation. So when the old math fails, executive hesitancy just naturally sets in.
SPEAKER_00They look at the numbers and panic.
SPEAKER_01Right. Leaders look at these massive upfront capital requirements. They fail to clearly plot the return on a standard timeline, and they just freeze.
SPEAKER_00So to understand why the old financial math is failing us, we first have to fundamentally redefine what it is we are actually buying when we authorize an AI budget.
SPEAKER_01Yeah, we have to rethink the whole purchase.
SPEAKER_00Aaron Powell Holman points out that this is a massive paradigm shift. We have to draw a hard line between traditional IT and artificial intelligence.
SPEAKER_01And that distinction is the foundational concept you have to grasp before you even open a spreadsheet.
SPEAKER_00Okay, let's break that down.
SPEAKER_01Well, traditional IT systems are deterministic, but artificial intelligence systems are probabilistic.
SPEAKER_00Aaron Powell Let's slow down and unpack those two terms because they completely change the trajectory of capital allocation. Deterministic versus probabilistic.
SPEAKER_01Yeah, it's a crucial difference.
SPEAKER_00With a deterministic system, if I am buying a like a massive ERP software suite, I know exactly how it operates. It is a closed logical loop. Right. I put a specific input into a predefined algorithm and I get a consistent predictable output.
SPEAKER_01And you get that exact same output every single time.
SPEAKER_00Exactly.
SPEAKER_01I mean if you input an employee's hours into an automated payroll system and apply the local tax rate, it spits out the exact paycheck amount. The physics of the system are known.
SPEAKER_00It's perfectly calculable.
SPEAKER_01Exactly. The output is entirely calculable and the project has a definitive end date. You install it, you test it, and it is finished.
SPEAKER_00But investing in AI is entirely different, right?
SPEAKER_01Oh, completely.
SPEAKER_00Because AI utilizes machine learning to infer patterns from massive data sets. That means its outputs come with varying degrees of certainty.
SPEAKER_01Right. It's guessing, essentially.
SPEAKER_00Yeah, it gives you the most likely answer or, you know, the most optimized path based on current data, which introduces this inherent unpredictability into both the performance and the final outcomes.
SPEAKER_01Unpredictability is really the defining characteristic here. The system is probabilistic because it's constantly calculating confidence intervals, not absolute truths.
SPEAKER_00And this is exactly the friction point Holman highlights for financial leaders, isn't it?
SPEAKER_01It is, that unpredictability is not a flaw in the software. It is a feature of learning and adaptation. I mean, an AI model must be continuously retrained with new data as market conditions shift or as customer behaviors change.
SPEAKER_00I look at traditional IT like buying a high-end calculator. You know exactly what it will do, it never changes its fundamental nature, and it never needs a pep talk.
SPEAKER_01Right. A calculator just calculates.
SPEAKER_00But investing in an enterprise AI system is much more akin to, well, it's like hiring a brilliant, highly compensated, but incredibly inexperienced intern.
SPEAKER_01That is a perfect analogy. Trevor Burrus, Jr.
SPEAKER_00Right. They learn, they adapt, and they might eventually run your entire supply chain. But their first few weeks or even months are going to be highly unpredictable.
SPEAKER_01They require constant management.
SPEAKER_00Exactly. And that intern analogy perfectly illustrates why traditional financial planning shatters when applied to AI.
SPEAKER_01Hiring that intern destroys the concept of a finished project timeline. Trevor Burrus, Jr.
SPEAKER_00Because they're always learning.
SPEAKER_01Right. Traditional IT investments rely heavily on clear implementation schedules and expected payback periods. You use standard capital budgeting techniques.
SPEAKER_00Like discounted cash flow analysis.
SPEAKER_01Exactly, DCF. You estimate the future cash flows the software will generate, you discount them back to today's present value, and boom, you get your ROI.
SPEAKER_00But how do I run a discounted cash flow analysis on an intern who is still figuring out the job?
SPEAKER_01You really can't.
SPEAKER_00If the AI model is experimental by nature, if it requires constant refinement, hyperparameter tuning, and ongoing data feeding, the timelines for when it actually starts generating that cash flow are incredibly vague.
SPEAKER_01Yeah, you can't put a firm date on an epiphany.
SPEAKER_00Right. You can't schedule exactly when the AI model will finally connect the dots and optimize your inventory carrying costs by 15%.
SPEAKER_01Which is exactly why forcing an AI initiative into a traditional CapEx model makes it look like a catastrophic investment on paper.
SPEAKER_00The math just doesn't work.
SPEAKER_01The initial costs appear astronomically high, the deployment timeline is totally fluid, and the payback period is just one big question mark.
SPEAKER_00And that leads directly into Holman's breakdown of the true costs of AI.
SPEAKER_01The denominator of our equation.
SPEAKER_00Exactly. To calculate the denominator in our ROI formula, the total investment, we have to completely rethink our cost category.
SPEAKER_01We really do.
SPEAKER_00So let's dive into that denominator. Holman insists that we have to split costs into two distinct buckets to understand the total cost of ownership.
SPEAKER_01Right. Hard investments and soft investments.
SPEAKER_00Hard investments, which are tangible, and soft investments, which are intangible. To ground this, let's imagine a fictional company. Let's call them Apex Logistics.
SPEAKER_01Okay, Apex Logistics.
SPEAKER_00They're a mid-market freight forwarder, and they're trying to implement a machine learning model to dynamically optimize their global shipping routes.
SPEAKER_01Perfect example. So for Apex Logistics, the hard investments are the tangible assets that usually make it onto the initial budget proposal.
SPEAKER_00The stuff you can easily write a check for.
SPEAKER_01Exactly. And Holman categorizes these into three main areas. First is data infrastructure. AI systems require massive volumes of both structured and unstructured data.
SPEAKER_00So it's not just buying some hard drives.
SPEAKER_01No, not at all. Apex isn't just paying for basic storage, they are paying for highly scalable databases, cloud environments, and the complex integration platforms required to pull data from everywhere.
SPEAKER_00Like their legacy systems, external weather feeds, global port authorities, all of it.
SPEAKER_01Aaron Powell Right. And it isn't just storing the data, it's computing it. The computational intensity of training a model is just staggering.
SPEAKER_00So that's the second hard cost.
SPEAKER_01Yeah. Computing resources. Training sophisticated algorithms requires high performance hardware, specifically cloud-based GPU clusters.
SPEAKER_00And the meter is constantly running on those cloud compute services during the training phase.
SPEAKER_01Oh, it's running fast. And it doesn't stop once the model is deployed either. Because inference, you know, the act of the AI actually making a decision in real time, that also requires significant compute.
SPEAKER_00Which brings us to the third hard cost, specialized talent.
SPEAKER_01This is a big one.
SPEAKER_00This is where I think many leaders severely underestimate the financial burden. I mean, in traditional IT, Apex Logistics might hire a consulting firm, they implement the routing software over six months, and then they leave.
SPEAKER_01It's a one-time capital expense.
SPEAKER_00Right. But with AI, because it is an evolving probabilistic system, you need data scientists and machine learning engineers permanently on the payroll.
SPEAKER_01And that recurring talent cost is a massive driver of the total cost of ownership. You cannot build a dynamic routing algorithm and then hand the keys over to a standard IT help desk.
SPEAKER_00They wouldn't know what to do with it.
SPEAKER_01No. You need specialized engineers to constantly monitor the model's outputs, tune the parameters, and manage the data pipelines.
SPEAKER_00So it is an ongoing operational expense, not a one-off capital expenditure.
SPEAKER_01Exactly. So Apex Logistics has their hard costs: complex data infrastructure, massive cloud compute bills, and a team of highly paid AI specialists.
SPEAKER_00It's a heavy lift, but it is comprehensible to a CFO.
SPEAKER_01Aaron Powell Sure, they can model that.
SPEAKER_00But then Holman introduces the soft investments. And this is where standard ROI calculations usually fall off the cliff.
SPEAKER_01Completely. Ignoring soft costs is perhaps the most fatal flaw in AI financial planning. Soft investments focus on the human and organizational side of the deployment.
SPEAKER_00Which means what? Exactly.
SPEAKER_01Things like continuous organizational learning, upskilling the current workforce, process re-engineering, and robust change management.
SPEAKER_00Aaron Powell Because implementing a system like this requires rethinking the entire business process at Apex Logistics. Exactly. I mean, the AI might suggest a radically different shipping route that looks totally counterintuitive to a veteran logistics manager. Trevor Burrus, Jr.
SPEAKER_01Right. They might think the machine is crazy.
SPEAKER_00Yeah. So you have to invest heavily in change management so the human operators actually trust and utilize the probabilistic outputs rather than just ignoring them and doing things the old way.
SPEAKER_01Aaron Powell I like to think about the hard costs as buying a state-of-the-art Formula One race car. The compute, the data, the engineers, that is the chassis, the engine, and the elite pit crew.
SPEAKER_00Okay, I follow you.
SPEAKER_01But the soft costs. That is repaving the entire racetrack and teaching every other driver on the circuit a completely new set of racing rules.
SPEAKER_00Aaron Powell That's a great way to put it.
SPEAKER_01Because you cannot draw a Formula One car onto a dirt track and expect to set a lap record. If Apex Logistics doesn't invest in redesigning their operational workflows to accommodate dynamic routing, the AI will just spin its wheels.
SPEAKER_00I want to highlight another critical soft cost Holman brings up here, which often gets swept under the rug during the excitement of an AI pitch.
SPEAKER_01What's that?
SPEAKER_00Ethical and regulatory compliance.
SPEAKER_01Oh yeah, that's huge.
SPEAKER_00Ensuring algorithmic transparency and complying with frameworks like GDPR or the EU AI Act takes sustained, dedicated resources.
SPEAKER_01It is a profound financial risk that's disguised as an IT problem.
SPEAKER_00Setting up internal accountability structures to understand exactly why an AI model made a specific decision that requires a massive operational lift.
SPEAKER_01It does. Think about it. If Apex's new algorithm inadvertently prioritizes routes that violate regional emissions regulations.
SPEAKER_00Or if it utilizes third-party data in a way that breaches privacy laws.
SPEAKER_01Exactly. The ensuing fines and reputational damage could easily wipe out years of theoretical ROI.
SPEAKER_00So investing softly in governance is really the only way to protect the hard investments.
SPEAKER_01It's essential insurance.
SPEAKER_00So when we look at the denominator of the ROI equation for Apex logistics, it is incredibly heavy.
SPEAKER_01Very heavy.
SPEAKER_00It requires ongoing specialized talent, massive compute, complete process re-engineering, and rigorous regulatory governance. Right. Having mapped out that complex cost structure, we have to look at the other side of the equation. The numerator, the returns.
SPEAKER_01And according to Holman, measuring the returns on AI is essentially an exercise in measuring the unmeasurable.
SPEAKER_00Which sounds encouraging for a CFO.
SPEAKER_01Right. The challenges compound significantly when we try to quantify the benefits. We touched briefly on the J curve of returns earlier.
SPEAKER_00Remind us about the J curve.
SPEAKER_01Well, Holman emphasizes that AI has notoriously long incubation periods. Initial deployments rarely produce immediate financial impact. The benefits tend to accrue incrementally at first.
SPEAKER_00As the model learns and the organization adapts.
SPEAKER_01Exactly. And then ideally they scale exponentially.
SPEAKER_00Which again breaks standard financial models like net present value and internal rate of return.
SPEAKER_01Completely breaks them.
SPEAKER_00If a CFO demands to see positive cash flow in quarter three and the AI model is still in its J-curve dip, the project gets killed prematurely.
SPEAKER_01Happens all the time.
SPEAKER_00So how do financial leaders actually justify the numerator if NPV doesn't work? Holman mentions option pricing models and real options analysis.
SPEAKER_01Yeah, real options.
SPEAKER_00I understand the concept of options in the stock market, you know, buying a call option.
SPEAKER_01Yeah.
SPEAKER_00But how does a CFO actually build an option pricing model for an enterprise software deployment?
SPEAKER_01It's a fascinating approach. It's borrowed directly from financial derivatives and it's gaining traction precisely because it addresses this uncertainty. Okay. Instead of viewing the AI initiative at Apex Logistics as a single rigid investment, the CFO evaluates it as a series of strategic options.
SPEAKER_00Meaning what?
SPEAKER_01Well, the initial massive spend on data infrastructure and talent isn't expected to yield immediate profit. Instead, that initial spend is viewed as the premium paid to purchase an option.
SPEAKER_00The option to do what exactly?
SPEAKER_01The option to rapidly deploy future use cases. By building a unified data lake and a core machine learning team today, Apex Logistics is purchasing the real option to deploy dynamic roading next year, and maybe an automated customer pricing algorithm the year after that.
SPEAKER_00So the model assigns financial value to the flexibility to expand, delay, or even abandon future projects based on how the underlying technology evolves.
SPEAKER_01Precisely. The CFO calculates the present value of the expected operating cash flows from those future hypothetical AI use cases and balances that against the cost of deploying them at scale later.
SPEAKER_00It essentially assigns a dollar value to strategic agility.
SPEAKER_01Exactly. It allows the board to understand that they aren't just buying a software tool. They are buying the foundational capability to compete in an AI-driven market.
SPEAKER_00That makes a lot of sense.
SPEAKER_01However, and this is a big however, Holman is quick to note that real options analysis introduces immense mathematical complexity. You can imagine. It requires a high degree of financial sophistication to execute without it devolving into, well, just speculative guesswork.
SPEAKER_00Right. You can't just make up numbers. And even if a CFO perfectly executes a real options model, there is a physical bottleneck that can destroy the entire calculation. Yes. Holman argues that data is the ultimate bottleneck. We hear that all the time, but usually it's framed with that tired cliche, you know, data is the new oil. Trevor Burrus, Jr.
SPEAKER_01Which is a terrible metaphor, honestly.
SPEAKER_00Why do you say that?
SPEAKER_01Because it implies data just needs to be extracted to be valuable. A better analogy would be refining raw uranium for a nuclear reactor.
SPEAKER_00Oh wow. Okay, raw uranium.
SPEAKER_01Think about it. The raw material is abundant and highly volatile, but it is completely useless and potentially dangerous until it undergoes an incredibly complex, expensive, and precise enrichment process.
SPEAKER_00Aaron Powell That's a powerful way to look at it. AI model performance is intrinsically tied to the quality, diversity, and volume of the data it is trained on.
SPEAKER_01Right. And the cost of data preparation, cleaning it, integrating siloed systems, and meticulously labeling it so the algorithm can process it is historically underbudgeted.
SPEAKER_00Leaders often assume that because AI is highly advanced, it can organically parse through messy, unstructured corporate data.
SPEAKER_01Aaron Ross Powell It cannot. It really can't. Poor data quality leads directly to inaccurate predictions. In our apex logistics scenario, if the historical shipping data is riddled with human entry errors.
SPEAKER_00Or old routing codes.
SPEAKER_01Yeah. The dynamic routing algorithm will confidently generate highly optimized yet completely incorrect shipping routes.
SPEAKER_00Garbage in, highly confident garbage out at a massive scale.
SPEAKER_01Exactly. And that leads directly to financial losses undermining the numerator of our ROI calculation before the system is even fully deployed.
SPEAKER_00So let's assume Apex Logistics actually sends the money to enrich their data. They refine the uranium. Okay. They survive the incubation period. The model works and starts optimizing routes. Now they have to quantify the benefits, which brings us to intangible returns.
SPEAKER_01The soft benefits.
SPEAKER_00Right. Holman states that AI delivers massive value that doesn't neatly align with traditional balance sheet categories.
SPEAKER_01Intangible benefits are critical, but they are notoriously difficult to isolate and monetize.
SPEAKER_00Like what? Give me an example.
SPEAKER_01Consider enhanced customer experience. If Apex Logistics uses their new AI capabilities to provide clients with hyper-accurate real-time predictions of freight arrivals.
SPEAKER_00Which clients would love.
SPEAKER_01Exactly. That transparency leads to higher customer satisfaction, which breeds brand loyalty, which ultimately increases customer lifetime value.
SPEAKER_00The logic is totally sound, but how does the CFO isolate the AI's specific role in that increased loyalty?
SPEAKER_01That's the million-dollar question.
SPEAKER_00Right. If a major client renews their contract for five years instead of three, was it because of the AI's predictive accuracy, or was it because Apex hired a charismatic new VP of sales?
SPEAKER_01Isolating that variable requires advanced analytics and quite frankly, a willingness to accept proxy metrics, which we will explore a bit later. Okay. Holman also highlights better strategic decision making as a core intangible benefit. AI surfaces insights that allow human leaders to make smarter, faster choices, contributing to a broader competitive advantage.
SPEAKER_00And furthermore, simply being recognized in the market as an AI innovator carries immense brand enhancement value.
SPEAKER_01It really does. It acts as a magnet for top-tier engineering talent and signals to investors that the company is future-proofing its operations.
SPEAKER_00But assigning a strict monetary value to brand enhancement or better decision making in a rigid ROI formula, I mean, it feels like trying to nail jelly to a wall.
SPEAKER_01It really does, which is why Hallman advocates for entirely new frameworks. But before we get to the solution, we have to address the most sobering part of his analysis.
SPEAKER_00Brace yourselves.
SPEAKER_01Yeah. Even if a company manages to perfectly calculate their data costs, successfully estimate their real options, and monetize their intangible benefits, there is a hidden force constantly working to erode the AI's value over time.
SPEAKER_00Section four of Holman's research explores this hidden erosion. It centers on three concepts: maintenance, technical debt, and a phenomenon known as model drift.
SPEAKER_01Model drift.
SPEAKER_00What exactly is model drift?
SPEAKER_01Model drift is the gradual degradation of an AI model's predictive accuracy. It is a major, often completely unbudgeted risk to sustained ROI.
SPEAKER_00How does a software model just degrade?
SPEAKER_01Well, it occurs because the real world is not static. When you train a machine learning model, you are training it on historical data. The model learns the underlying patterns of that specific snapshot in time. But as time moves forward, external factors inevitably change, consumer preferences shift, macroeconomic conditions fluctuate, and unexpected global events occur.
SPEAKER_00Like a pandemic.
SPEAKER_01Exactly. So the fundamental patterns that the model relies on to make decisions are actively changing underneath it.
SPEAKER_00The model doesn't break in the way a traditional software application crashes.
SPEAKER_01No, no error screens.
SPEAKER_00It simply becomes increasingly out of touch with reality. It continues to make highly confident predictions based on how the world used to behave.
SPEAKER_01Let's bring back the Formula One analogy. Okay. With traditional software, technical debt is like a leaky pipe in the garage. It's annoying, you know, you need to fix it, but it's an isolated problem. Sure. But model drift is like the race car itself. Every lap you run, the tires degrade. Furthermore, the track layout is actively shifting due to weather conditions.
SPEAKER_00The track is moving.
SPEAKER_01Yep. Yes. If you don't spend millions of dollars bringing the car into the pit stop for new tires, and if you don't constantly update the driver on the changing track conditions, the car will eventually crash.
SPEAKER_00That is a phenomenal way to visualize it. Preventing that crash requires massive ongoing operational expenses just to maintain the model's day one baseline performance.
SPEAKER_01Exactly. Organizations must budget for continuous infrastructure costs, energy consumption, support services, and most importantly, the specialized personnel required to continuously retrain the model with fresh data.
SPEAKER_00Aaron Powell Which raises a very uncomfortable question for a business leader.
SPEAKER_01Aaron Powell Extremely uncomfortable.
SPEAKER_00How much of my annual AI budget needs to be permanently ring fenced purely to prevent degradation rather than to create new value? I mean if I am a CEO, I want my capital funding new features and aggressive innovations. I do not want to spend millions of dollars every year just to keep the AI from getting dumber.
SPEAKER_01It is a bitter pill for executives to swallow. While Holman doesn't prescribe a universal percentage, he makes it emphatically clear that establishing permanent governance frameworks to monitor for drift is non-negotiable.
SPEAKER_00You just have to do it.
SPEAKER_01You do. Furthermore, this is where technical debt becomes catastrophic.
SPEAKER_00How so?
SPEAKER_01Well, if a company rushes the initial development phase to get an AI product to market quickly, they inevitably create technical debt or fragile architectures.
SPEAKER_00It's just like taking out a high interest loan to finish the project on time.
SPEAKER_01Yes. And with AI, the interest rate on that technical debt compounds aggressively.
SPEAKER_00Because the model is always running.
SPEAKER_01Right. Subaldemal architecture makes retraining the model significantly more difficult and expensive. It accelerates the deterioration process and leads to massive future rework costs that completely obliterate the initial ROI projections.
SPEAKER_00And the stakes for this deterioration are incredibly high. If a streaming service's recommendation engine suffers from model drift and suggests the wrong movie, the cost is minimal.
SPEAKER_01You watch a bad movie, no big deal.
SPEAKER_00Right. But as AI becomes deeply integrated into core operational workflows, model drift at Apex Logistics could mean mispricing tens of millions of dollars in freight contracts.
SPEAKER_01Or optimizing routes that suddenly violate new international shipping sanctions.
SPEAKER_00Exactly. As models are granted more autonomy over critical business decisions, their unseen degradation transitions from an IT annoyance to a severe financial and legal liability.
SPEAKER_01Continuous governance and ethical accountability are permanent fixtures of the AI lifecycle, and they continuously draw down the net ROI over time if they are not accurately forecasted in the initial budget.
SPEAKER_00Okay, so when we aggregate all of this, the probabilistic nature of the system, the massive hard and soft costs, the vague incubation periods, the reliance on perfectly enriched data, the elusive intangible returns, and the continuous financial bleed of model drift, the picture is incredibly bleak for anyone clinging to a standard spreadsheet.
SPEAKER_01It's terrifying, honestly.
SPEAKER_00Holman's solution to this paralysis is radical but necessary. He proposes that we throw out the old math entirely and utilize a completely modified formula.
SPEAKER_01Right, because the traditional formula is dangerously simplistic. It calculates net profit divided by the cost of investment.
SPEAKER_00Which assumes the investment is a fixed static event.
SPEAKER_01And that the profit is a clearly definable immediate cash flow.
SPEAKER_00It is clean, it is comforting, and for AI, it is entirely inadequate. So Holman asks us to take a completely different view of the balance sheet with his modified ROI formula.
SPEAKER_01Walk us through it.
SPEAKER_00First, you take every benefit, your tangible cost savings, and your monetized intangible revenue boosts, and you bundle them together as the numerator.
SPEAKER_01Okay, got it.
SPEAKER_00Then you subtract your entire cost burden. But here is the critical shift. You must include the initial capital expenditure and the ongoing operational maintenance as one unified cost figure.
SPEAKER_01Right, you can't hide the maintenance.
SPEAKER_00Exactly. Finally, you divide that net gain by that same unified cost figure.
SPEAKER_01By isolating ongoing cost as a permanent, undeniable fixture in the denominator, and by explicitly demanding the quantification of intangible benefits in the numerator, this formula forces a culture of absolute honesty in the boardroom.
SPEAKER_00It mathematically forces the executive team to admit that the new intern is going to require expensive perpetual training.
SPEAKER_01While simultaneously giving them a framework to champion the long-term strategic value that intern brings to the organization's culture and capabilities. But introducing a new mathematical formula is only useful if leadership teams know how to wield it without succumbing to institutional biases.
SPEAKER_00Very true. And Holman outlines five common dysfunctions traps that well-intentioned organizations fall into when evaluating these investments. Let's go through them.
SPEAKER_01The most prevalent dysfunction is approaching AI with a static mindset. Organizations calculate their ROI assuming the model will deliver a flat 10% efficiency gain in perpetuity.
SPEAKER_00They fail to conduct rigorous sensitivity or scenario analysis.
SPEAKER_01Exactly. They build the financial model without asking. What happens to our ROI if a major competitor launches a superior AI tool next quarter?
SPEAKER_00Or what happens if the European Union alters their data privacy regulations and we lose access to 20% of our training data?
SPEAKER_01Or what happens when model drift accelerates faster than anticipated? By failing to model out those changing market conditions and technological variables, organizations risk wildly overestimating their returns.
SPEAKER_00Aaron Powell And completely blinding themselves to operational risks.
SPEAKER_01Exactly.
SPEAKER_00The second dysfunction Hallman identifies is ignoring the fundamentally experimental nature of artificial intelligence. How does this manifest in corporate strategy?
SPEAKER_01It typically manifests as executive hubris.
SPEAKER_00Hubris, okay.
SPEAKER_01Yeah. A leadership team reads a glowing analyst report and dictates a global, enterprise-wide rollout of an AI platform on day one.
SPEAKER_00Aaron Powell Just flip the switch everywhere.
SPEAKER_01Right. But AI development requires rigorous hypothesis testing, continuous iteration, and learning from failure. If an organization applies rigid linear project management methodologies like the classic waterfall approach, it suffocates the exploration required to actually tune the model.
SPEAKER_00Aaron Powell You cannot mandate an epiphany on a GATT chart.
SPEAKER_01You really can't. Deploying an under-evaluated, unpiloted solution at scale is a guaranteed recipe for massive financial destruction.
SPEAKER_00Which leads directly into the third trap. Treating AI projects in isolation.
SPEAKER_01Oh, the silos.
SPEAKER_00This is particularly rampant in large siloed enterprises. The marketing department secures budget for a generative AI content tool. The supply chain team builds a predictive inventory model.
SPEAKER_01And human resources purchases a third-party AI screening platform.
SPEAKER_00And none of these departments are communicating.
SPEAKER_01Which means they are duplicating massive data infrastructure costs, utilizing conflicting governance standards, and entirely failing to leverage any collective data synergies across the enterprise.
SPEAKER_00It fractures the capital investment and misaligns the technology with the broader strategic goals of the company.
SPEAKER_01It's a huge waste of resources. The fourth trab Holman highlights is one that plagues almost every traditional board of directors, an overwhelming over-emphasis on cost savings.
SPEAKER_00I want to explore this deeply because focusing primarily on cost reduction overlooks the broader transformative value of AI-like revenue expansion, new product development, and revolutionized customer experiences.
SPEAKER_01It's a very narrow view.
SPEAKER_00It is. But that sounds incredibly inspiring. But let's look at the reality of corporate finance. Cost cutting is the primary language most boards speak. If a CTO is pitching a $10 million AI infrastructure build, the board's immediate reaction is to look for $10 million in headcount reductions to offset it. How does a leader practically pitch intangible innovation to a skeptical board?
SPEAKER_01It is arguably the most difficult conversation a technology leader will have, but the argument has to be framed around long-term survival. Using AI purely for cost reduction is a race to the bottom. Let's look at Apex Logistics again. If they use an AI chatbot to automate their customer service and fire 50% of their support staff, their short-term ROI for that specific quarter will look phenomenal.
SPEAKER_00The CFO will be thrilled.
SPEAKER_01Right. But if that chatbot provides a rigid, frustrating experience that cannot handle complex logistical problems, the clients will eventually leave for a competitor who offers better service.
SPEAKER_00So an overemphasis on cost savings is essentially mortgaging your future market share for a temporary spike on the quarterly earnings report.
SPEAKER_01Precisely. You are chronically underinvesting in the customer experience for a short-term accounting win. AI's true power lies in its ability to do things the organization couldn't previously do at all, not merely doing old things slightly cheaper.
SPEAKER_00That's a powerful distinction.
SPEAKER_01The final dysfunction is inadequate risk assessment. If an organization fails to fully account for technical, operational, and regulatory risks, their ROI calculation is pure fiction.
SPEAKER_00Technical risks include scalability, right?
SPEAKER_01Yeah. Will the model hold up when 100,000 users query it simultaneously instead of 100?
SPEAKER_00And operational risks encompass the massive disruptions to existing workflows during the integration phase. And as we discussed, regulatory risks can introduce catastrophic legal liabilities.
SPEAKER_01If financial probabilities are not assigned to those risks and factored into the denominator of the modified formula, the organization is flying blind.
SPEAKER_00So we have established that the math is complex, the costs are pervasive, the returns are elusive, the system continuously degrades, and the boardroom is filled with strategic traps.
SPEAKER_01It's a lot.
SPEAKER_00It paints an incredibly daunting picture for any executive trying to champion an AI initiative. But Holman transitions from diagnosing the problem to providing a concrete solution. He outlines an executive playbook of best practices to actually execute this modified formula and secure sustainable value.
SPEAKER_01This playbook is essential because it translates all the theoretical complexity we have unpacked into actionable strategic execution.
SPEAKER_00Let's examine the playbook. The first strategy is to adopt a staged approach.
SPEAKER_01Which is the direct antidote to the global rollout trap we just talked about.
SPEAKER_00Right. A staged approach mandates the use of controlled pilots and proofs of concept to aggressively manage risk and validate assumptions before any scaled deployment occurs.
SPEAKER_01It is about linking capital deployment to empirical evidence. Holman advocates for incremental investment models tied to specific performance milestones, utilizing agile development practices.
SPEAKER_00This structure mirrors venture capital financing.
SPEAKER_01It really does.
SPEAKER_00By utilizing a staged approach with incremental funding, a CFO is essentially treating internal AI deployments the same way a VC treats a startup ecosystem.
SPEAKER_01They provide seed funding for a localized pilot.
SPEAKER_00Exactly. If that pilot proves value, the team unlocks Series A funding to deploy it across one business unit. If it scales successfully, they secure Series B funding for enterprise integration. It establishes a financial safety net.
SPEAKER_01It forces the AI initiative to continuously justify its own existence before consuming more capital, entirely mitigating the risk of a massive upfront sunk cost.
SPEAKER_00The second strategy in the playbook is shifting the focus entirely to outcome-oriented metrics.
SPEAKER_01The board does not care about the technical elegance of the model.
SPEAKER_00Right. They do not care if it uses a novel transformer architecture or has a trillion parameters. They only care if it reduces supply chain latency by three days or increases gross margins by 2%.
SPEAKER_01So technology leaders must define clear key performance indicators that are intimately tied to overarching business goals. Holman suggests utilizing balanced scorecards to track both financial returns and non-financial operational improvements.
SPEAKER_00He also emphasizes value stream mapping.
SPEAKER_01Yes, value stream mapping is crucial here. Organizations shouldn't just inject AI everywhere to seem innovative, like sprinkling AI dust on everything. Right, that doesn't work. They need to analyze their entire operational workflow, identify the exact operational bottlenecks causing friction, and deploy AI surgically to alleviate those specific choke points where it will deliver maximum measurable impact.
SPEAKER_00The third piece of the playbook is incorporating both hard and soft ROI, which requires a mechanism to monetize those intangible benefits. We discussed the difficulty of assigning a dollar value to things like brand enhancement or customer satisfaction.
SPEAKER_01This is where proxymetrics become invaluable.
SPEAKER_00What does an effective proxy metric actually look like in the real world?
SPEAKER_01Well, a proxymetric is a quantifiable data point that stands in for an unmeasurable qualitative concept. You cannot measure the emotion of customer delight on a spreadsheet. Right. But you can measure a reduction in average customer service resolution time, or an increase in the frequency of repeat purchases, or a measurable uptick in your net promoter score.
SPEAKER_00You utilize those hard observable metrics as proxies for the soft benefit. Then, using techniques like contingent valuation, a financial team can estimate the monetary value of, for example, a one-point increase in that net promoter score based on historical revenue data.
SPEAKER_01Holman also advocates for utilizing total economic impact or TEI frameworks.
SPEAKER_00How does a TEI framework differ from standard ROI?
SPEAKER_01A TEI framework is a much more holistic financial methodology. It provides a highly structured way to evaluate not just the direct costs and benefits, but also the strategic flexibility, those real options we explored earlier, and the specific risk factors over a multi-year horizon.
SPEAKER_00Okay, that makes sense.
SPEAKER_01Crucially, building a TEI model requires deep cross-functional engagement. If the IT department builds the financial model in a vacuum, they will inevitably fail to account for the operational soft costs that the human resources or operations departments would foresee.
SPEAKER_00The fourth strategy is continuous monitoring and optimization, which is the direct operational defense against model drift.
SPEAKER_01Organizations must implement comprehensive performance dashboards that provide real-time insights into the model's accuracy. A data science team needs to know the exact moment a model's predictive accuracy drops from 95% to 91%.
SPEAKER_00So you aren't waiting until quarter end to find out it's broken.
SPEAKER_01Exactly. This requires establishing rigorous model management protocols, strict version control, mandatory retraining schedules, and continuous validation pipelines.
SPEAKER_00It requires treating the AI not as a static tool, but as a living asset that requires continuous medical checkups and rehabilitation to perform at its peak.
SPEAKER_01And a critical component of that monitoring is establishing feedback loops from the human operators.
SPEAKER_00Right, the people actually using it.
SPEAKER_01The employees interacting with the AI on the front lines are almost always the first to notice when its outputs begin to misalign with reality. Their qualitative feedback is an indispensable early warning system for model drift.
SPEAKER_00The final strategy in Holman's playbook solves the trap of isolated projects, strategic alignment and portfolio management. Organizations must stop treating AI as a series of ad hoc experiments and start managing it as a cohesive, centralized portfolio.
SPEAKER_01Investments must be prioritized based on their strategic fit with the company's ultimate competitive goals, not just allocated to whichever department shouts the loudest during budget season.
SPEAKER_00Treating AI like an isolated standalone project is akin to a company buying a single lottery ticket and hoping it secures their market dominance. Strategic portfolio management means treating AI like a diversified mutual fund. You balance high-risk, high-reward foundational investments with lower-risk immediate optimization tools.
SPEAKER_01You diversify the technological risk, centralize the data infrastructure to eliminate redundancies, and govern the entire portfolio holistically.
SPEAKER_00Centralized governance ensures accountability, regulatory compliance, and ethical standards are uniformly applied across the entire enterprise. That is the only path to ensuring AI drives meaningful, sustainable, and legally sound results.
SPEAKER_01That's right.
SPEAKER_00So as we synthesize everything we have unpacked today, the overarching message is clear. AI is not traditional IT.
SPEAKER_01Not even close.
SPEAKER_00It is an experimental, probabilistic, data-hungry, long-term strategic play. Accurately measuring its return on investment requires organizations to completely abandon static financial models and adopt dynamic, highly flexible frameworks.
SPEAKER_01We must rigorously account for both the tangible costs of compute and the massive intangible costs of change management.
SPEAKER_00We must anticipate and financially budget for the perpetual erosion of model drift. And we must align our capital investments into a cohesive, rigorously governed portfolio utilizing incremental milestone-based funding.
SPEAKER_01The stakes for mastering this new financial paradigm could not be higher.
SPEAKER_00It really is do or die.
SPEAKER_01The organizations that successfully implement Holman's comprehensive, honest approach to ROI measurement are the ones that will actually capture the unprecedented transformative potential forecasted by McKinsey and Goldman Sachs. They will build insurmountable competitive advantages and drive sustainable growth, while their competitors remain paralyzed, staring at broken spreadsheets. I love this part.
SPEAKER_00Throughout this deep dive, we have continuously highlighted how artificial intelligence fundamentally differs from a traditional piece of software. It is not deterministic, it learns, it adapts to its environment, it requires continuous training, careful management, strict ethical governance, and it actively degrades in performance if you ignore it. Yes. Frankly, the operational profile of an AI system sounds significantly less like a piece of enterprise software and much more like a human employee.
SPEAKER_01The parallels to managing a human workforce are undeniable, particularly when considering the necessity of continuous education and the risks of organizational misalignment.
SPEAKER_00Consider the implications of that shift. So here is something to mull over with your executive team as you map out your strategic initiatives for the next decade. As artificial intelligence models become increasingly autonomous, as they take on vastly more complex cognitive tasks and become deeply inextricably integrated into the core functioning of your business. Yeah. Will there come a day when we stop trying to awkwardly force AI onto the balance sheet as a capital expenditure and instead begin accounting for it on the PL as digital labor?
SPEAKER_01It is a profound reframing of how we value the fundamental engines of corporate productivity.
SPEAKER_00When the diagnostic tools are failing to capture the reality of the patient, it is time to build new tools. Thank you for joining us for you listening on this deep drive to the true cost and strategic value of artificial intelligence. Keep demanding the broader context behind the math, and as always, stay insanely curious.