By Multi-agent AI systems do not behave like chatbots. They loop hundreds of times across tools, APIs, external data sources, and multiple models before delivering a result. Every hop in that loop adds latency, and in a centralized architecture, that latency compounds exponentially. For physical AI, autonomous systems, and real-time commerce, the resulting delays are not a performance inconvenience. They are a system failure.
In this interview on TFiR, Jon Alexander, SVP of Product for the Cloud Technology Group at Akamai, breaks down why centralized cloud infrastructure is the wrong architecture for production agentic AI, what enterprises are actually missing when they focus only on GPU capacity, and how distributed inference solves the end-to-end latency problem at scale.
Guest: Jon Alexander, SVP of Product for the Cloud Technology Group at Akamai
Show: TFiR
Here is what every platform engineer and AI infrastructure architect needs to know.
Technical Deep Dive
Q: Why is latency becoming the defining infrastructure problem for production AI?
Jon Alexander, SVP of Product for the Cloud Technology Group at Akamai, explains that the AI workload landscape is undergoing a fundamental shift. Early AI deployments focused on training large foundational models in centralized infrastructure, then moved into relatively contained applications like chatbots and customer support. The transition happening now is toward real-time, pervasive AI where agents handle tasks previously owned by algorithms, applications, or humans, and those agents must make decisions at machine speed, not human speed. That shift makes latency a hard architectural constraint, not just a performance metric.
“Needing to make decisions at machine speed, not at human speed is the key transition that we’re seeing.” — Jon Alexander, SVP of Product for the Cloud Technology Group, Akamai
Q: What actually breaks when multi-agent workflows are forced through centralized cloud architecture?
Alexander draws a direct parallel to microservices architecture: an agentic workflow is composed of multiple tools, multiple models, and multiple interaction steps that must all be coordinated to produce an outcome. Unlike a single-shot question-and-answer interaction, an agent may loop 100 to 200 times, calling APIs, pulling data, generating and executing code at each step. When users, data, and the third-party systems the agent calls are all distributed, routing all of that through a single centralized location compounds latency at every iteration and degrades end-to-end performance in proportion to the number of loop cycles.
“A centralized system is not going to be the optimal location to run all of those components of the solution.” — Jon Alexander, SVP of Product for the Cloud Technology Group, Akamai
Q: Is GPU capacity and token generation speed the right thing to optimize for in agentic AI?
Alexander confirms that GPUs are expensive, constrained, and important, but argues the focus on GPU throughput is incomplete for agentic systems. In real-world agentic deployments, up to 90% of total execution time is not spent waiting for the GPU to generate tokens. It is spent on CPU-bound work: tool calling, writing to the file system, executing code, and coordinating with external systems. Optimizing GPU speed improves one component while the majority of execution time remains unaddressed. The end-to-end latency in these systems is governed by orchestration across CPUs, GPUs, and external connectivity, not by token generation rate alone.
“Up to 90% of the overall execution time isn’t on the GPU. It’s actually running on CPUs where it’s doing this other work.” — Jon Alexander, SVP of Product for the Cloud Technology Group, Akamai
Q: What are the biggest infrastructure friction points as organizations move to multi-step, multi-agent systems pulling from APIs and knowledge bases?
Alexander frames the problem as a dimensionality shift: chatbots were essentially one-dimensional, while real-world agents operate in a multi-dimensional space requiring tool access, file system interaction, code execution, and coordination across models running on different cloud providers and AI labs. Forcing that multi-dimensional coordination through a centralized location means every iteration of the agent loop carries compounded latency. A loop iteration that takes 100 milliseconds run 100 times produces 10 seconds of end-to-end duration. For a human chatbot interaction that may be tolerable, but for physical AI, robotics, or autonomous vehicles, 10 seconds is a task failure.
“If this is an autonomous vehicle, 10 seconds is a lifetime. That task is going to fail.” — Jon Alexander, SVP of Product for the Cloud Technology Group, Akamai
Q: Is the AI infrastructure challenge today comparable to the early web, and does the CDN model apply?
Alexander draws a direct parallel between 2026 and the web in 1996: early adopters are demonstrating potential, but pervasive deployment has not yet occurred. Akamai was founded to solve what was then called the worldwide wait, the problem of getting content from centralized servers to distributed users when physical Internet capacity was insufficient. Alexander argues the same structural problem exists now with AI inference. The core technologies Akamai developed for that problem, routing, load balancing, and workload placement, are directly relevant to distributed AI orchestration. The difference is that AI optimization requires multiple dimensions simultaneously: performance, cost, reliability, resilience, compliance, and security.
“I think we’re at that same point. This is where we are today in 2026 is equivalent to the web in 1996.” — Jon Alexander, SVP of Product for the Cloud Technology Group, Akamai
Q: How should enterprises rethink their architecture to prepare for distributed AI execution?
Alexander recommends a crawl-walk-run framework. The crawl phase is about confirming the agent solves a real business problem and establishing the right guardrails and controls before scaling. The walk phase focuses on readiness for production: handling real load, hardening security, and validating the system for real-world deployment. The run phase is where performance optimization becomes the priority, defining clear SLOs, establishing performance monitoring benchmarks, and architecting specifically to hit those targets. Alexander positions this final phase as where Akamai’s distributed infrastructure can help enterprises scale out and scale up to meet those performance objectives reliably.
“Make sure you’ve got really clear performance monitoring benchmarks that you’re trying to target and then you’ve architected to achieve those.” — Jon Alexander, SVP of Product for the Cloud Technology Group, Akamai
Resources & Documentation
- Akamai, cloud and distributed infrastructure platform for AI inference, content delivery, and edge computing
- Akamai Blog, technical analysis and market perspective on AI infrastructure and distributed inference trends
***
👇 Click to Read Full Raw Transcript
Swapnil Bhartiya: While it’s true that the biggest challenge in AI is securing GPUs, which are hard to find, but when it comes to production, the real bottleneck is latency. But as AI moves from simple chatbots that answer simple questions into complex agentic systems that take actions on your behalf autonomously, centralized cloud architectures are starting to choke. When AI agents coordinate and make real time decisions, latency becomes an architectural constraint. And the future of production AI requires a totally different approach. It requires a distributed approach. And to explain this, today, we have with us once again John Alexander, SVP of Product for the Cloud Technology Group at Akamai. John, it’s great to have you on the show.
Jon Alexander: Great swap. Yeah, good to be back on the show. Great to chat again.
Swapnil Bhartiya: Let’s talk about latency. Why is it becoming the defining infrastructure problem for production AI, especially as enterprises transition from simple chatbots to complex agentic systems that work autonomously?
Jon Alexander: That’s right, yeah. No, and I think that’s the key point. Like we’re on the cusp of a significant change in the types of AI workloads that our customers are looking to deploy. And so again, if we think back a couple of years, a lot of the focus was on developing the model. So big training clusters were being developed, aggregating huge amounts of data into centralized infrastructure to create these amazingly powerful foundational models. Then we started to shift into application of AI into relatively simple workloads. Simple in terms of the application, powerful in terms of the capabilities that are being deployed, but primarily things like chatbots, customer support, relatively simple sort of one dimensional interactions that end users were having with AI. Today, one of the most prominent uses for AI that we see is for coding. So we’re starting to get into much more sophisticated use cases. But again, deployed in a relatively constrained environment. What we’re seeing our customers talking about now is moving into more real time AI where it becomes pervasive throughout their business process. And they’re looking at deploying agents to handle many of the tasks that previously were handled by other algorithms and applications that they had, or even humans. And so they’re looking at a much broader range of use cases. And this is where inference is evolving and becoming more real time. And needing to make decisions at machine speed, not at human speed is the key transition that we’re seeing when multi
Swapnil Bhartiya: agent workflows are forced through traditional centralized cloud architecture. What actually starts to break as we
Jon Alexander: move from relatively simple question, answer type interactions with AI and we’re moving into agentic workflows, you’re moving into a multi step looping process where the agent is making multiple requests to the model, and that is a multi step process. I personally think my background is in Internet architecture, so I think about this as a lot like a microservices type deployment where an application is built with decomposed into all these different components and all of these pieces have a separate purpose that needs to be coordinated to achieve a specific business outcome. In an agent you have kind of a similar concept where you have multiple components of the system, different tools, different models that you’re talking to, and different interactions at different steps of the process which are all required to achieve the outcome. So it’s not a single shot where I just fire off one response, get one answer and deliver that back to the user. I go around maybe 100 times, maybe 200 times, I call these different systems, I pull in data from APIs, I generate code that I might execute. All of these are multiple steps and depending upon where these are running, it can compound the latency. Now, a centralized cloud can work well if all of the data, if all of the users, if the models, if everything is located close. Often what we’re finding is that isn’t the case. Users are distributed, data is distributed, and often the systems that you’re talking to, the APIs that you’re calling, the tools that you’re calling, these third party systems, they’re distributed as well. And this all pulls us back to a centralized system is not going to be the optimal location to run all of those components of the solution. And so in these multi agent workflows, we’re often finding that they don’t work well. For real world applications where distribution of data, of users and the tools exist.
Swapnil Bhartiya: No matter who you talk to, the conversation always comes back to GPU capacity and token generation. Is that focus right or is it incomplete? What are organizations actually missing and what should they be focused on as we shift from generating answers to taking actions?
Jon Alexander: This is one of the kind of big items that we’ve identified as we’re seeing customers deploying real world agentic applications. GPUs are incredibly important. Like they’re one of the most expensive and constrained parts of the architecture. So capacity there is limited. You want to make sure you’re using it effectively. But the way that we’re seeing these agentic applications built a large portion of the execution duration, waiting for the GPU to generate tokens, to generate the response. It’s actually in the tool calling. So the agent is calling out to third party systems. It’s writing files to the local file system, it’s executing code, so it’s doing all of this other work. And that can be up to 90% of the overall execution time isn’t on the GPU. It’s actually running on CPUs where it’s doing this other work. And so you definitely see a lot of benefit from the GPUs being fast. That can be a bolt on it. But increasingly we’re seeing that the end to end latency is being controlled by all these other parts of the agentic system. And so orchestrating that and coordinating between the CPUs and the GPUs, strategically making sure that you’re leveraging that infrastructure in the right locations, making sure you’ve got connectivity to external systems, all of that is as important, if not more important, for improving the performance in these real world scenarios.
Swapnil Bhartiya: What are the biggest friction points organizations face in today’s infrastructure as they move from simple question answer to multi step, multi agent system pulling from APIs, data sources and knowledge bases?
Jon Alexander: I think you hit on the key point. We’re moving from relatively simple, almost one dimensional problems into a multi dimensional space. All of the use cases that you talked about, this is what we see customers starting to build. And again, we’re at the very beginning of the real world adoption of agents like these complex systems. But there’s a disconnect what those real world agents need. So they need the ability to call tools, they need access to a file system, they need to be able to execute code, they need to be able to coordinate across different models, running on different cloud providers or on different AI labs. All of that needs a coordination of infrastructure that needs to be distributed. And so putting or trying to force all of that into one central location is not the optimal architecture. It means that you’re compounding the latency on all of these iterations, all of the different sequences in that flow, and your end to end duration goes up exponentially. So if one iteration on the loop takes 100 milliseconds, you’ve got a loop that takes 100 iterations, that’s 10 seconds. So you’ve got 10 seconds end to end. Now, if this is a human talking to a chatbot, maybe we’re willing to wait 10 seconds. If this is a physical AI, if this is a robot trying to make a decision in real time, if this is an autonomous vehicle, 10 seconds is a lifetime, it’s way too long. And like that, that task is going to fail. And so this is the fundamental disconnect that we’re seeing with real world agents, their expectations are exponentially different than traditionally. What we’ve seen with chatbots, where it’s been a more human centric, human latency tolerance.
Swapnil Bhartiya: Today’s AI landscape reminds me of the early days of the web. Big promise, real performance problems. And that’s exactly the problem Akamai solved back then. By inventing CDN and bringing content closer to users. Are we at the same inflection point when it comes to AI? And if so, how should we rethink where workloads should run closer to users and data, or they should be centralized?
Jon Alexander: Yeah, I think there’s a lot of parallels to the evolution of the web. So the web was the killer application for the Internet. And if we rewind 30 years now I think we’re at that same point. This is where we are today in 2026 is equivalent to the web in 1996. Right. At the very early stage, I think the early adopters are seeing the potential, they’re building the experiences, but we haven’t achieved that sort of pervasive deployment. I think that’s what’s coming. And this is very much kind of a problem that Akamai set out to solve 30 years ago. Solving what at that time we called the worldwide wait. Getting capacity from these or getting content from these central servers to users was a challenge. There just wasn’t the physical capacity in the Internet to achieve that. And therefore Akamai solved this through creating the content delivery network. And then we were able to help those sites deliver the experiences that today are common and kind of expected. I think there’s very similar transformation that’s coming today with AI, we’re moving from this phase of experimentation, early, early adoption, and now we’re starting to move into real commercialization and the technology starting to become pervasive and really solving business problems. So chatbots were powerful. They were great proofs of concept for the technology, but they didn’t transform the business economics or change kind of the fundamental capabilities of companies. I think the opportunity that we’re seeing going forward as people are deploying agents, they can fundamentally change how the business operates, their ability to innovate, the scale they can operate. There’s lots of potential benefits. And so that’s what we’re seeing is many orders of magnitude more adoption as we look forward now. How do we achieve that? This is the challenge, I think, if you follow the tech press, I read far too many stories every single day about new data centers being deployed, kind of supply chain constraints, scarcity of memory limitations, in the overall supply to meet the demand that’s coming. And that’s where I think Akamai sees an opportunity for us to help scale out the adoption of inference. And so part of this is making sure that we can achieve the kind of ubiquitous availability of inference, make sure that we’re able to deliver the performance expectations that the customers expect. And so the real world latency budget that people have, like, that’s a fundamental requirement that we need to be able to meet. But we also need to make sure that it’s cost effective, that it’s reliable, resilient, it can meet the kind of compliance and security requirements that customers have. That’s multiple dimensions of optimization. It needs a slightly different set of technologies than Akamai originally kind of identified with CDN, but some of them are very similar. And like one of the pieces, maybe just to kind of finish with is one of the core problems that we see is this idea of orchestration across the inference platform. And so we believe that the solution is to leverage infrastructure that is in the right location with the right capabilities for the right workload. That doesn’t mean that everything needs to be distributed. Some pieces can be centralized, but some pieces do need to be close to either where the data is, where the GPUs are, or even where the end user is. And so optimizing for placement of workloads, scheduling between CPUs and GPUs to handle those handoffs of like the tool calling, make sure you’re not leaving GPUs idle as you’re waiting to pull data from third parties, or you’re executing code locally to generate a response. That’s a big optimization problem that is very, very similar to a lot of the early work that Akamai did around routing and load balancing, placement of workloads, core technology that really helped us scale out the early stages of the Internet that we think is going to be very relevant here. So similar problems that are very close to the DNA of Akamai, infrastructure providers like Akamai continue to evolve.
Swapnil Bhartiya: How should enterprises rethink their own architecture to prepare for the shift towards distributed AI execution?
Jon Alexander: I think it’s a crawl, walk, run. I mean, typically what we’re seeing is people are experimenting, they’re building the application. And I think one of the key things is focus on are you solving a real problem? Like there’s a lot of agents that don’t end up delivering value and therefore confirm that you’re really solving a business problem. Make sure you’ve got the right guardrails, make sure you’ve got the right controls in place. That’s the first thing. So that’s the crawl phase. Then the walk phase is like, hey, how are you going to scale this out? Make sure that this is able to handle the load that’s coming, make sure that’s ready for real world deployment, make sure it’s secure. Like that’s key. So that’s the walk phase. The run phase though, is where you get into the optimization around performance and make sure it’s meeting the expectations. Again, back to some of the examples that we talked about. So for physical AI, for an autonomous vehicle, even for commerce, there are expectations that end users have or that systems have, and if you’re not meeting those expectations, the system’s going to fail, it’s not going to meet the business objectives. And so that’s the final level of like, hey, make sure that you’re actually achieving the SLOs. Make sure you’ve got really clear performance monitoring benchmarks that you’re trying to target and then you’ve architected to achieve those. And that’s really where Akamai can help you scale out and scale up the performance to meet those objectives.
Swapnil Bhartiya: John, thank you so much for joining us today and sharing these insights on latency and what it really means for the future of AI infrastructure in production. For everyone else watching, please make sure to check out Akamai and their blog to see how this market is evolving and how you should be preparing. John, thank you so much for your time and I look forward to chat with you again.
Jon Alexander: Great, thank you very much. Cheers.
