By Enterprise AI deployments are running into a hard ceiling: centralized inference clusters cannot deliver the sub-second response times that AI agents require, and GPU utilization collapses every time a workload pauses to call a tool or retrieve remote data. Data center supply is constrained, memory is scarce, and stacking more centralized capacity does not fix a placement and scheduling problem.
In this interview on TFiR, Jon Alexander, SVP of Product, Cloud Technology Group at Akamai, breaks down why the infrastructure architecture that works for model training fails for inference at scale, and what distributed orchestration actually needs to look like.
Guest: Jon Alexander, SVP of Product, Cloud Technology Group at Akamai
Show: The Agentic Enterprise
Watch the discussion on YouTube
Here is what every platform engineer and AI infrastructure architect needs to know.
Technical Deep Dive
Q: Why does centralized AI inference fail to meet real-world latency requirements at scale?
Jon Alexander, SVP of Product, Cloud Technology Group at Akamai, argues that the infrastructure problem facing AI inference in 2026 is structurally identical to the problem that broke centralized web servers in 1996: the physical capacity of the network and the central compute layer cannot keep pace with user demand. Latency budgets for interactive AI agents are non-negotiable, and centralized inference clusters simply cannot be close enough to users, data sources, or execution environments to meet them consistently. Alexander frames this not as a temporary supply shortage but as a fundamental placement problem.
“The real world latency budget that people have, that’s a fundamental requirement that we need to be able to meet.”
Jon Alexander, SVP of Product, Cloud Technology Group, Akamai
Q: How is the AI infrastructure challenge in 2026 comparable to the early web in 1996?
Alexander draws a direct parallel between today’s AI adoption curve and the state of the web thirty years ago. Early adopters in 1996 could see the potential of the web but pervasive deployment had not yet arrived; the same is true for AI in 2026. Akamai’s original answer to the “World Wide Wait” was to distribute content delivery through a CDN, moving capacity closer to users rather than scaling central servers. Alexander argues the solution structure for AI inference is analogous: distribute workloads to where data, GPUs, and users actually are, rather than concentrating everything in a handful of hyperscale regions.
“This is where we are today in 2026 is equivalent to the web in 1996, at the very early stage.”
Jon Alexander, SVP of Product, Cloud Technology Group, Akamai
Q: Why do AI agents demand a fundamentally different infrastructure architecture than chatbots?
Chatbots demonstrated the technology but did not change business economics. Agents operate differently: they call tools, retrieve data from external systems, execute code locally, and chain multiple model calls together. Each of those steps introduces latency and leaves GPUs sitting idle while the agent waits for a response from a third party. At the scale of enterprise agent deployment, that idle GPU time compounds into a serious cost and performance problem. Alexander describes this as the shift from proof-of-concept to workloads that “fundamentally change how the business operates.”
“Chatbots were powerful, they were great proofs of concept for the technology, but they didn’t transform the business economics or change the fundamental capabilities of companies.”
Jon Alexander, SVP of Product, Cloud Technology Group, Akamai
Q: What causes GPU idle time in AI agent workloads and how does scheduling fix it?
When an AI agent calls an external tool, queries a database, or executes local code to generate a response, the GPU handling that inference job sits idle until the external call returns. At high request volumes this idle time is a direct drain on GPU utilization and inference economics. Alexander identifies CPU-to-GPU scheduling and intelligent handoff orchestration as the mechanism to reclaim that wasted capacity: route the waiting work off the GPU, fill the GPU slot with another ready task, and return the result when the external call completes. This is directly analogous to the routing and load-balancing techniques Akamai developed for web traffic.
“Make sure you’re not leaving GPUs idle as you’re waiting to pull data from third parties or you’re executing code locally to generate a response, that’s a big optimization problem.”
Jon Alexander, SVP of Product, Cloud Technology Group, Akamai
Q: When should AI inference workloads be distributed versus kept centralized?
Alexander is explicit that distributed does not mean everything runs at the edge. The decision framework is workload-specific: centralize what does not have strict latency or data-proximity requirements, and distribute what does. Placement decisions should be driven by where the data lives, where the GPUs are available, and how close the workload needs to be to the end user to meet its latency budget. The orchestration layer is responsible for making those placement decisions dynamically, not statically at design time.
“The solution is to leverage infrastructure that is in the right location with the right capabilities for the right workload. That doesn’t mean that everything needs to be distributed.”
Jon Alexander, SVP of Product, Cloud Technology Group, Akamai
Q: What are the non-latency requirements that distributed AI inference infrastructure must also satisfy?
Alexander lists cost efficiency, reliability, resilience, and compliance as requirements that must be satisfied alongside latency. He frames these as multiple simultaneous dimensions of optimization, not a simple trade-off. Enterprise buyers cannot accept a distributed inference architecture that meets latency targets but fails on data sovereignty, security posture, or uptime. This multi-dimensional constraint is part of what makes the orchestration problem hard and why a single hyperscale region cannot be the default answer.
“We need to make sure that it’s cost effective, that it’s reliable, resilient, it can meet the kind of compliance and security requirements that customers have.”
Jon Alexander, SVP of Product, Cloud Technology Group, Akamai
Resources & Documentation
- Akamai Cloud Technology Group, distributed cloud infrastructure and AI inference platform from Akamai
***
👇 Click to Read Full Raw Transcript
Swapnil Bhartiya: Today’s AI landscape reminds me of the early days of the web. Big promise, real performance problems. And that’s exactly the problem Akamai solved back then. By inventing CDN and bringing content closer to users. Are we at the same inflection point when it comes to AI? And if so, how should we rethink where workloads should run closer to users and data, or they should decentralized?
Jon Alexander: Yeah, I think there’s a lot of parallels to the evolution of the web. So the web was the killer application for the Internet. And if we rewind 30 years now, I think we’re at that same point. So this is where we are today in 2026 is equivalent to the web in 1996. Right. At the very early stage, I think the early adopters are seeing the potential, they’re building the experiences, but we haven’t achieved that sort of pervasive deployment. I think that’s what’s coming. And this is very much kind of a problem that Akamai set out to solve 30 years ago. Solving what at that time we called the world Wide wait. Getting capacity from these or getting content from these central servers to users was a challenge. There just wasn’t the physical capacity in the Internet to achieve that. Therefore, Akamai solved this through creating the content delivery network. And then we were able to help those sites deliver the experiences that today common and kind of expected. I think there’s very similar transformation that’s coming today with AI. We’re moving from this phase of experimentation, early adoption, and now we’re starting to move into real commercialization and the technology starting to become pervasive and really solving business problems. So chatbots were powerful. They were great proofs of concept for the technology, but they didn’t transform the business economics or change kind of the fundamental capabilities of companies. I think the opportunity that we’re seeing going forward, as people are deploying agents, they can fundamentally change how the business operates, their ability to innovate, the scale they can operate. There’s lots of potential benefits. And so that’s what we’re seeing is many orders of magnitude more adoption as we look forward now. How do we achieve that? This is the challenge. I think if, if you follow the kind of the tech press, I read far too many stories every single day about new data centers being deployed, kind of supply chain constraints, scarcity of memory, limitations in the overall supply to meet the demand that’s coming. And that’s where I think Akamai sees an opportunity for us to help scale out the Adoption of inference. And so part of this is making sure that we can achieve the kind of ubiquitous availability of inference, make sure that we’re able to deliver the performance expectations that the customers expect. And so, like, the real world latency budget that people have, like, that’s a fundamental requirement that we need to be able to meet. But we also need to make sure that it’s cost effective, that it’s reliable, resilient, it can meet the kind of compliance and security requirements that customers have. That’s multiple dimensions of optimization. It needs a slightly different set of technologies than Akamai originally identified with cdn, but some of them are very similar. And one of the pieces, maybe just to finish with is one of the core problems that we see is this idea of orchestration across the inference platform. We believe that the solution is to leverage infrastructure that is in the right location with the right capabilities for the right workload. That doesn’t mean that everything needs to be distributed. Some pieces can be centralized, but some pieces do need to be close to either where the data its, where the GPUs are, or even where the end user is. And so optimizing for placement of workloads, scheduling between CPUs and GPUs to handle those handoffs of the tool, calling, make sure you’re not leaving GPUs idle as you’re waiting to pull data from third parties or you’re executing code locally to generate a response. That’s a big optimization problem that is very, very similar to a lot of the early work that Akamai did around routing and load balancing, placement of workloads, core technology that really helped us scale out the early stages of the Internet that we think is going to be very relevant here. So similar problems that are very close to the DNA of, of Akamai.
