By Multi-agent AI systems coordinate across tools, file systems, code execution environments, and multiple models running on different cloud providers. Every coordination hop adds latency. In a loop with 100 iterations, even 100 milliseconds per step produces 10 seconds of end-to-end delay. For chatbots, that is marginal. For autonomous vehicles or physical AI making real-time decisions, it is a hard failure.
In this interview on TFiR, Jon Alexander, SVP of Product, Cloud Technology Group at Akamai, breaks down the infrastructure friction points organizations face as they move from simple question-and-answer systems to multi-step, multi-agent architectures pulling from APIs, data sources, and knowledge bases.
Guest: Jon Alexander, SVP of Product, Cloud Technology Group at Akamai
Show: TFiR
Here is what every platform engineer and AI infrastructure team needs to know.
Technical Deep Dive
Q: What are the biggest infrastructure friction points for multi-agent AI systems moving beyond simple chatbots?
Jon Alexander, SVP of Product, Cloud Technology Group at Akamai, explains that the shift from single-turn question-and-answer interactions to multi-step agent workflows introduces a multidimensional infrastructure problem. Real-world agents need to call tools, access file systems, execute code, and coordinate across models running on different cloud providers and AI labs. None of that maps cleanly onto the centralized infrastructure patterns enterprises built for earlier AI workloads.
“We’re moving from relatively simple, like almost one dimensional problems into a multidimensional space.”
Jon Alexander, SVP of Product, Cloud Technology Group, Akamai
Q: Why does centralized infrastructure compound latency in multi-agent AI loops?
Alexander describes a straightforward compounding effect: when each iteration in an agent loop incurs a fixed latency penalty, forcing all coordination through a single central location stacks those penalties sequentially. A loop with 100 iterations at 100 milliseconds per step produces 10 seconds of end-to-end latency. Distributed infrastructure reduces per-hop cost by placing coordination closer to where each model, tool, or data source actually runs.
“Putting or trying to force all of that into one central location is not the optimal architecture. It means that you’re compounding the latency on all of these iterations, all of the different sequences in that flow, and your end to end duration goes up exponentially.”
Jon Alexander, SVP of Product, Cloud Technology Group, Akamai
Q: When does multi-agent AI latency become an outright failure rather than just a performance issue?
Alexander draws a clear line between human-centric use cases and physical AI. A person interacting with a chatbot may tolerate a 10-second response. A robot making a real-time decision or an autonomous vehicle responding to road conditions cannot. For those systems, 10 seconds is not a degraded experience; the task fails entirely. This distinction marks the fundamental shift in requirements as AI moves from software-only assistants into physical and autonomous systems.
“If this is an autonomous vehicle, 10 seconds is a lifetime, it’s way too long. And that task is going to fail.”
Jon Alexander, SVP of Product, Cloud Technology Group, Akamai
Q: How is multi-agent AI latency tolerance different from traditional chatbot latency tolerance?
Alexander frames it as a mismatch between the expectations enterprises carried over from chatbot-era infrastructure and the reality of agentic workloads. Chatbots were designed around human response-time tolerance: seconds are acceptable. Real-world agents operating in physical environments or tight feedback loops operate on tolerances measured in milliseconds. The infrastructure assumptions built for one do not transfer to the other, and that gap is where most current deployments run into problems.
“Their expectations are exponentially different than traditionally what we’ve seen with chatbots, where it’s been a more human centric human latency tolerance.”
Jon Alexander, SVP of Product, Cloud Technology Group, Akamai
👇 Click to Read Full Raw Transcript
Swapnil Bhartiya: What are the biggest friction points organizations face in today’s infrastructure as they move from simple question answer to multi step, multi agent system pulling from APIs, data sources and knowledge bases?
Jon Alexander: I think you hit on the key point. We’re moving from relatively simple, like almost one dimensional problems into a multidimensional space. Like all of the use cases that you talked about. This is what we see customers starting to build. And again, we’re at the very beginning of the real world adoption of agents like these complex systems. But there’s a disconnect what those real world agents need. So they need the ability to call tools, they need the access to a file system, they need to be able to execute code, they need to be able to coordinate across different models, running on different cloud providers or on different AI labs. All of that needs a coordination of infrastructure that needs to be distributed. And so putting or trying to force all of that into one central location is not the optimal architecture. It means that you’re compounding the latency on all of these iterations, all of the different sequences in that flow, and your end to end duration goes up exponentially. So if one iteration on the loop takes 100 milliseconds, you’ve got a loop that takes 100 iterations, that’s 10 seconds. So you’ve got 10 seconds end to end. Now, if this is a human talking to a chatbot, maybe we’re willing to wait 10 seconds. If this is a physical AI, if this is a robot trying to make a decision in real time, if this is an autonomous vehicle, 10 seconds is a lifetime, it’s way too long. And like that, that task is going to fail. And so this is the fundamental disconnect that we’re seeing with real world agents. Their expectations are exponentially different than traditionally what we’ve seen with chatbots, where it’s been a more human centric human latency tolerance.
