By When AI moves from a controlled experiment to a live application serving real users, the infrastructure requirements shift in ways that raw GPU capacity cannot address. Inference is continuous, not a one-time training run, which means compute, storage, and networking must all be architected together for sustained performance and cost control. Teams that treat production AI as simply an extension of their experimentation environment run into latency failures, runaway costs, and data pipeline gaps that more hardware does not fix.
In this interview on TFiR, Danielle Cook, Director of Product Marketing at Akamai, walks through the specific infrastructure decisions architects must make when moving AI from experimentation to production at scale.
Guest: Danielle Cook, Director of Product Marketing at Akamai
Show: TFiR
Here is what every platform architect and infrastructure engineer running AI in production needs to know.
Technical Deep Dive
Q: What changes about infrastructure requirements when AI moves from experimentation to production?
Danielle Cook, Director of Product Marketing at Akamai, explains that experimentation focuses narrowly on whether the model produces acceptable output. Production introduces real users, real latency requirements, real data, and real billing pressure. Because inference is continuous rather than a discrete training job, the entire stack including compute, storage, and networking becomes a factor in application performance, not just the GPU layer.
“In production you have real users, real latency requirements, real data, real bills that are associated with this. And because inference is continuous, that’s where you start realizing that GPUs are only part of the architecture.”
Danielle Cook, Director of Product Marketing, Akamai
Q: When should AI workloads use CPUs instead of GPUs?
Cook notes that not every component of an AI application requires GPU resources. Tool calls, API integrations, and database interactions can run effectively on CPUs, which changes the cost calculus significantly. Architects who map workload types to the right compute tier avoid over-provisioning GPUs and reduce operational cost without sacrificing performance where it matters.
“GPUs, I need it for this stuff, but actually tools, tool calls, APIs, databases, all of that can actually largely work with my CPUs.”
Danielle Cook, Director of Product Marketing, Akamai
Q: How does semantic caching reduce cost and improve AI application performance?
Cook identifies semantic caching as a core architectural lever for production AI. By storing and serving semantically equivalent prompt-response pairs, teams avoid redundant inference calls, which directly reduces GPU utilization and cost. The benefit extends to the customer experience: the right answers reach users faster and more consistently without repeated round-trips to the model.
“You have to be considering how you’re going to use semantic caching so that the right prompts, the right answers, the right experience gets to your customers.”
Danielle Cook, Director of Product Marketing, Akamai
Q: How should architects think about compute placement for AI applications?
Cook frames compute proximity to users as a first-class architectural decision, not a network optimization afterthought. Real-time, interactive AI applications generate continuous data in motion, and placing compute closer to end users reduces latency across the inference pipeline. Architects must plan for both edge compute placement and the data movement patterns that connect inference to the customer experience.
“You need to be putting compute close to your users, but you also need to be thinking about the core data movement around inference, around AI, about the customer experience.”
Danielle Cook, Director of Product Marketing, Akamai
Q: Does moving to production AI require a complete infrastructure rethink or just incremental adjustments?
Cook is direct: architects should evaluate the entire infrastructure, not make isolated adjustments. AI applications are real-time and interactive, which means data is continuously in motion across the stack. Budget sustainability, model routing decisions, semantic caching strategy, and compute placement must all be planned together, making production AI an infrastructure challenge that spans every layer.
“AI does change everything. It is real time, it’s interactive, it is the data moving all around. How do you build for that while maintaining it, making sure it’s cost effective?”
Danielle Cook, Director of Product Marketing, Akamai
Resources & Documentation
- Akamai, cloud computing, security, and content delivery platform with infrastructure designed for AI and real-time application workloads
***
👇 Click to Read Full Raw Transcript
Swapnil Bhartiya: When enterprises move from experimenting with AI to running it in live applications, what actually changes? Why does production AI put different demands on infrastructure as compared to running traditional applications?
Danielle Cook: You know, in production you tend to focus on or in experimentation, rather, you tend to focus on the model. Like, does it work? Is my output good? But in production now, you have real users, real latency requirements, real data, real bills that are associated with this. And because inference is continuous, it’s not a model that you train and walk away from. That’s where you start realizing that GPUs are only part of the architecture. You need compute, you need storage, you need networking, all of that to actually determine how your application performs. And you can’t just throw more GPUs at this at every step. So you know your infrastructure is changing. You also might be looking at it going, okay, so GPUs, I need it for this stuff, but actually I need, you know, tools, tool calls, APIs, databases, all of that can actually largely work with my CPUs. So choosing the right model might be part of an exciting decision for you, but making everything work around it at production scale is the infrastructure problem that organizations are having to solve.
Swapnil Bhartiya: How much is this changing how architects approach infrastructure? Is it a minor adjustment or do they need to completely rethink their infrastructure strategy?
Danielle Cook: Architects should be looking at the the entire infrastructure, right? AI does change everything. It is real time, it’s interactive, it is the data moving all around, complete data in motion. So how do you build for that while maintaining it, making sure it’s cost effective? How do you not blow your entire budget trying to deliver an amazing experience to your customers? When you’re architecting for it, you have to be considering things like model route, you have to be considering how you’re going to use semantic caching so that the right prompts, the right answers, the right experience gets to your customers. And so, yes, it’s an infrastructure challenge. You need to be putting compute close to your users, but you also need to be thinking about the core data movement around inference, around AI, about the customer experience.
