By AI training dominates budget conversations, but inference is the cost that accrues on every single model response in production. Most enterprise AI budgets were built around GPU procurement and model development, leaving the ongoing operational cost of inference unplanned and underestimated. Token pricing, the figure most vendors quote, does not capture the full picture once architecture, distribution, and scale enter the equation.
In this interview on TFiR, Ari Weil, VP of Product Marketing at Akamai, breaks down why AI inference must be budgeted as a distributed system and how architectural decisions, not just model selection, determine the real cost of running AI at scale.
Guest: Ari Weil, VP of Product Marketing at Akamai
Show: TFiR
Here is what every platform engineer and AI infrastructure team needs to know.
Technical Deep Dive
Q: Why does AI inference get less attention than training even though it drives ongoing cost?
Ari Weil, VP of Product Marketing at Akamai, explains that training captures headlines because it drives massive data center deals, large capital investments, and is the centerpiece of IPO narratives for AI-focused companies. The focus on who is building the largest data center for the next multi-billion or multi-trillion weight model has made training the dominant frame for AI investment discussions. Inference, by contrast, is the cost that arrives after the excitement of training and accumulates every time a model responds to a real request in production.
“Inference is the bill that you get after you get all excited about training and it arrives every single time your model takes a breath.” — Ari Weil, VP of Product Marketing, Akamai
Q: What exactly triggers an inference cost in a production AI system?
Weil draws a clear line between the training lifecycle, which includes pre-training, post-training, and fine-tuning, and inference, which is everything that happens when the model is deployed and responding to real-world requests. Every time the model’s intelligence is called upon outside of a training context, an inference event occurs and a cost is incurred. This makes inference a continuous, volume-driven operational expense rather than a one-time capital investment.
“Inference is what happens when somebody needs your intelligence to be deployed.” — Ari Weil, VP of Product Marketing, Akamai
Q: Why is budgeting AI inference as a GPU line item or per-application cost a mistake?
Weil argues that treating AI spend as either a GPU procurement budget or a per-application cost fails to account for the distributed system complexity that governs real inference workloads. The architecture spanning where models run, how requests are routed, and how compute is distributed is what ultimately determines the bill. Teams that only track token price or GPU spend without modeling the full system architecture will encounter costs they did not plan for.
“You need to budget like building a distributed system, because the architecture is what ultimately is going to decide your bill.” — Ari Weil, VP of Product Marketing, Akamai
Q: Why does token pricing alone fail to predict the real cost of running AI in production?
Token price, the per-token cost vendors quote for generating model outputs, is a single variable in a much larger cost equation. Weil emphasizes that while token pricing is commonly cited, it does not capture the architectural factors that drive actual spend at scale: where the model runs, how traffic is distributed, latency requirements, and how the inference layer integrates with the rest of the system. Teams conversant only in token pricing are missing the structural cost drivers.
“You’re going to hear people quote token price, but the architecture is what ultimately is going to decide your bill.” — Ari Weil, VP of Product Marketing, Akamai
Q: What prior engineering disciplines should teams draw on when designing AI inference architecture?
Weil points directly to cloud native and edge native system design as the relevant prior art for AI inference architecture. Large organizations already have familiarity with building distributed systems, and representative architectures exist in the market that can be applied. Teams should take their cues from how cloud native and edge native systems have been designed and operated over the past several years rather than treating AI inference as an entirely new engineering problem.
“Take some of your cues from building cloud native systems and, to some extent, edge native systems for the last several years.” — Ari Weil, VP of Product Marketing, Akamai
Q: How does where you run AI inference change the cost and outcome?
Weil frames the location of inference execution as a primary cost and performance variable, not a secondary infrastructure detail. Whether inference runs centrally in a hyperscale cloud, distributed across edge nodes, or in a hybrid configuration affects latency, data transfer costs, and ultimately the economics of serving requests at scale. The decision of where to run inference is an architectural choice with direct financial consequences that teams need to evaluate explicitly.
“Where you run it changes everything.” — Ari Weil, VP of Product Marketing, Akamai
Resources & Documentation
- Akamai, cloud computing, security, and content delivery platform with AI inference infrastructure capabilities
***
👇 Click to Read Full Raw Transcript
Swapnil Bhartiya: And today we have with us Ari Weil, VP of Product Marketing at Akamai, to break down the real economics of AI inference and why where you run it changes everything.
Ari Weil: Absolutely. So, first of all, it’s great to be back on the program with you again, and I’m glad to have a chance to talk about the cost side of the equation when we think about AI applications and workloads. And I think to start off with, I would suggest that training is really the headline that everybody loves. It’s driving these massive data center deals, big investments, and some of the circular investment that we’ve seen coming out of anything from chip manufacturers and card manufacturers to the major hyperscale clouds that are also building out their own infrastructure and procuring it from others. And even if we talk about the landscape of companies that are looking to do their first IPO, a lot of the focus has been on how large of a data center they’re building out to build the next multi billion, multi trillion weight model. I think that training and that headline sort of factor is real. But if we think about it, inference is the bill that you get after you get all excited about training and it arrives every single time your model takes a breath, every time that you’re not training, either pre training or post training or fine tuning that model. Inference is what happens when somebody needs your intelligence to be deployed. And so I think the highlight from our perspective, where we would start a conversation, is to say you need to stop budgeting your AI applications as line items to do with just the GPUs that you’re procuring or just the applications that you’re building you need to budget like building a distributed system, which is something that many large organizations have familiarity with, or at least there are representative architectures for in the marketplace, because you’re going to hear people quote token price, how much does it cost per token for me to generate a workload? But the architecture is what ultimately, ultimately is going to decide your bill. And that’s the part that people need to get more conversant on and take some of their cues from building cloud native systems and to some extent, edge native systems. For the last several years.
