By Most AI deployments look strong in controlled environments and collapse under real user load. Token budgets exceed projections within days, latency degrades as concurrent users grow, and infrastructure choices made early create lock-in that blocks expansion into new markets. These are not edge cases. They are the predictable outcome of committing to inference deployments without evaluating infrastructure against production criteria.
In this interview on TFiR, Jon Alexander, SVP of Product for the Cloud Technology Group at Akamai, walks through the specific infrastructure requirements enterprises must evaluate before committing to large-scale, long-term AI inference deployments.
Guest: Jon Alexander, SVP of Product for the Cloud Technology Group at Akamai
Show: TFiR
Here is what every platform engineer and AI infrastructure architect needs to know.
Technical Deep Dive
Q: What should enterprises evaluate before committing to large-scale AI inference deployments?
Jon Alexander, SVP of Product for the Cloud Technology Group at Akamai, argues that the buying criteria for AI infrastructure closely mirror established cloud buying criteria, with portability and flexibility as the starting point. Enterprises need to avoid tying their applications to a single vendor’s technology stack and should build on cloud-native technologies like Kubernetes to preserve deployment flexibility. The goal is to retain the ability to expand into new markets and move workloads without penalty.
“A lot of the cloud buying criteria are as important here. Think carefully about portability. Think about how you can actually have flexibility around where you deploy applications.” — Jon Alexander, SVP of Product for the Cloud Technology Group, Akamai
Q: Why do AI applications that work in demos fail when user load increases?
Alexander identifies elasticity as a critical and frequently underestimated infrastructure requirement. Applications routinely perform well in single-user or low-concurrency scenarios and begin to degrade visibly at 100 users, then more severely at 1,000 or 10,000. The architectural decisions that support a demo do not automatically scale, and teams that over-provision capacity upfront face cost problems while teams that under-provision face performance collapse. The architecture must support incremental, low-friction scaling from the start.
“A lot of applications work great for that demo. And then as soon as you’ve got 100 users, 1,000 users, 10,000 users, it starts to look very, very different.” — Jon Alexander, SVP of Product for the Cloud Technology Group, Akamai
Q: How does vendor lock-in in storage and databases affect AI deployment flexibility?
Alexander frames AI fundamentally as a data problem and a context management problem, making storage and database portability a core infrastructure concern rather than a secondary consideration. Locking AI workloads into a database or storage infrastructure that is only accessible through a single provider restricts where applications can run and creates egress penalties when teams need to move data. Ensuring that the right context can be loaded into memory from wherever the application is deployed is essential for long-term architectural flexibility.
“You don’t want to be tied into a database or a storage infrastructure that can only be accessed through one provider.” — Jon Alexander, SVP of Product for the Cloud Technology Group, Akamai
Q: What are the real cost risks of large-scale AI inference deployments?
Alexander points to token budget overruns as a widely reported and actively occurring problem, where teams exhaust their entire token allocation within days due to uncontrolled usage patterns. Cost management is a concrete operational requirement that must be built into the deployment architecture, not handled reactively. Alongside raw cost, teams need to define a performance budget that specifies latency targets and determine how those targets will be maintained as the deployment scales beyond a single location or server.
“There are a lot of horror stories going around at the moment about people blowing their entire token budget in a few days with uncontrolled costs.” — Jon Alexander, SVP of Product for the Cloud Technology Group, Akamai
Q: How should teams think about latency targets when scaling AI inference?
Alexander frames latency as a performance budget that must be defined early and tracked as scale increases. Many teams begin with simple single-location or single-server deployments where latency is naturally low, but fail to plan for what happens when the application is adopted more broadly across distributed user bases. Defining specific latency targets upfront and designing the architecture to achieve those targets at scale, not just at launch, is a requirement that needs to be addressed before the deployment grows.
“Think about the performance budget that you have. What are the latency targets that you need to hit as this scales out, and how are you going to achieve that?” — Jon Alexander, SVP of Product for the Cloud Technology Group, Akamai
Resources and Documentation
- Akamai, cloud computing, security, and content delivery platform referenced throughout as the infrastructure context for this discussion
- Kubernetes, open-source container orchestration system, explicitly recommended for providing deployment flexibility across environments
***
👇 Click to Read Full Raw Transcript
Swapnil Bhartiya: Before enterprises commit to large scale long term inference deployments, what should they be expecting and demanding from their AI infrastructure?
Jon Alexander: It’s an important question. And so like generally I think some of the kind of cloud buying criteria are as important here. And so think carefully about sort of portability. Think about how you can actually have flexibility around where you deploy applications. You don’t want to tie into one thing, vendor’s technology, stack running on top of cloud, native technologies like Kubernetes, obviously important that gives you flexibility around where you deploy. I think making sure that you’ve got elasticity. Think about how it’s going to scale up. Often a lot of applications work great for that demo and then as soon as you’ve got 100 users, 1,000 users, 10,000 users, it starts to look very, very different. And eventually when it’s wildly successful and you’ve got tens of billions of users, that’s when things can get really challenging. And so think about how can you have that architecture that will scale. So you don’t want to be provisioning a huge amount of capacity upfront, but you want to have flexibility to grow. You want to have the ability to expand into new markets and you want to be able to do that with low friction. You don’t want to be tied into what infrastructure that’s going to give you penalties for moving the data. So again, a lot of the way I think about this is AI is really a data problem. It’s a context management problem. It’s making sure you got the right context at the right time loaded into memory. But having that available to wherever you want to run the application is super important. So you don’t want to tie into a database or a storage infrastructure that can only be accessed through through one provider. And then I think last piece is think about the performance. We talked a lot here about latency. Cost is an important dimension. A lot of people think about managing costs. A lot of horror stories going around at the moment about people blowing their entire token budget in a few days with kind of uncontrolled costs. Obviously that’s an important factor. But thinking about the performance budget that you have, what are the latency targets that you need to hit as the scales out and how are you going to achieve that? Because a lot of people are starting again, just simple deployments. One location may be running on a local machine or just on a single server. Everything works great when it’s one user. How’s it going to work when it’s much more widely adopted? That’s kind of a key area to think about. How do you scale out to achieve that performance?
