By Agentic AI systems are not chatbots. They execute multi-stage, multi-turn inference loops, call external tools and APIs, interact with memory systems, run sandboxed code, and pull context from distributed real-world endpoints. Every round-trip in that loop carries latency, and centralized cloud infrastructure compounds that latency on every iteration. For real-time systems, the math fails fast.
In this interview on TFiR, Jon Alexander, SVP of Product for the Cloud Technology Group at Akamai, breaks down why the architectural assumptions behind centralized cloud create a hard ceiling for agentic AI workloads and what practitioners need to understand before deploying agents in production.
Guest: Jon Alexander, SVP of Product for the Cloud Technology Group at Akamai
Show: TFiR
Here is what every platform engineer and AI infrastructure architect needs to know.
Technical Deep Dive
Q: Why did centralized cloud infrastructure work for early AI workloads but struggle with agentic AI?
Jon Alexander, SVP of Product for the Cloud Technology Group at Akamai, explains that early AI infrastructure investment was concentrated on training, which required large coherent GPU clusters connected by high-speed interconnects to share memory across tightly coupled compute. When inference arrived, the first workloads were simple chatbots that could tolerate centralized deployment because latency requirements were loose and compute density in one location provided economies of scale. Agentic AI changes the equation entirely: agents are multi-stage, multi-turn systems that call tools, loop repeatedly through inference cycles, and interact with distributed real-world endpoints, which centralized architecture was never designed to serve efficiently.
“We’re at the cusp now of the next wave of adoption where we’re moving into the rise of agents.” — Jon Alexander, SVP of Product for the Cloud Technology Group, Akamai
Q: What architectural constraints made centralized GPU clusters the right choice for model training?
Alexander describes training infrastructure as optimized around a single constraint: memory. Large clusters of coherent GPUs were connected with high-speed interconnects so that GPUs could share available memory across the cluster, enabling the high-performance computing environment needed to produce large foundational models. This tight coupling and co-location made sense when the workload was a single, long-running training job with no real-world interaction requirement. The constraint driving architecture was memory bandwidth and coherence, not geographic distribution or latency to external systems.
“Memory was really the constraint that we’re optimizing for, and so we create these very closely connected clusters to achieve that high performance computing infrastructure.” — Jon Alexander, SVP of Product for the Cloud Technology Group, Akamai
Q: What does an AI agent actually look like architecturally, and why does that structure create distributed infrastructure requirements?
Alexander breaks an agent into its real components: the agent logic itself runs on CPU infrastructure and calls out to one or more models running on GPU infrastructure, which may include multiple providers such as OpenAI, Anthropic, or self-hosted open source models on separate GPU clusters. Beyond the models, agents are increasingly dominated by the tools they call: memory systems, file system access, sandboxed code execution, third-party APIs, and MCP endpoints. Each of those tool targets exists at a different physical location. Forcing all of that interaction through a single centralized region means every tool call crosses unnecessary geographic distance.
“Increasingly what we’re seeing is the agents are dominated by the tools that they’re calling.” — Jon Alexander, SVP of Product for the Cloud Technology Group, Akamai
Q: How does latency compound across agentic inference loops and at what point does it become a production problem?
Alexander uses a direct calculation to frame the problem: adding 100 milliseconds of latency per loop iteration across 100 loop iterations produces 10 seconds of added latency. For a human-facing chatbot, 10 seconds may be tolerable. For physical AI systems or any real-time application, 10 seconds is operationally unacceptable. Because agentic systems are defined by their looping behavior, latency does not add linearly across a single request the way it does in a stateless API call. It multiplies, and centralized infrastructure that places agent logic far from the tools it calls accelerates that multiplication.
“Even adding 100 milliseconds of latency, if you’re looping 100 times, that’s 10 seconds of latency that you’ve added. For physical AI, for any type of real time system, 10 seconds is way too long. It’s a lifetime.” — Jon Alexander, SVP of Product for the Cloud Technology Group, Akamai
Q: What do enterprises actually observe when they force agentic workloads through centralized infrastructure?
Alexander notes that the surface symptoms enterprises see are performance degradation, higher bills, and runaway token usage, but the underlying cause is architectural. The real world is not centralized: users, devices, APIs, and the decisions agents must make are distributed. When every agent interaction with a distributed real-world system is routed through a single central location, compounding latency is the structural result, not an edge case. Enterprises often do not connect the symptom to the cause because the failure appears as a cost or performance problem rather than an architectural mismatch.
“The real world isn’t centralized. Data, users, devices, the APIs and all the decisions you’re making, they’re distributed.” — Jon Alexander, SVP of Product for the Cloud Technology Group, Akamai
Q: Why is physical geography now a hard constraint in agentic AI architecture rather than a secondary consideration?
Alexander frames geography as a first-class architectural constraint because agents do not operate in isolation. They interact with the real world through tool calls, API calls, MCP endpoints, and external systems, none of which are co-located in a single data center. Not all users are physically close to a centralized region. Not all third-party systems route efficiently through a central hub. Because the agent loop repeats many times per task, geographic distance between the agent and its tool targets is not amortized across a single request. It is paid on every iteration, making proximity between compute and the systems it calls a direct performance variable.
“Physical geography is becoming a really important constraint in these architectures.” — Jon Alexander, SVP of Product for the Cloud Technology Group, Akamai
Resources & Documentation
- Akamai, cloud and edge infrastructure platform referenced throughout as the solution context for distributed AI inference
***
👇 Click to Read Full Raw Transcript
Swapnil Bhartiya: We have spent decades moving everything to the cloud and it solved a lot of problems. But when it comes to AI inferencing specifically, it demands a fundamentally different architectural approach. Why is the centralized cloud model not working for Agent Ki?
Jon Alexander: It’s a good point. And I mean the cloud has been massively transformational for many enterprises. So the ability to rent infrastructure versus own infrastructure, the ability to consume infrastructure as a service versus as infrastructure, these have been transformational. But when you think about AI, we’re going through a big transformation in terms of the types of applications that we’re seeing running on top of the infrastructure. So a couple of years ago a lot of focus was on training models and so a lot of the discussion and architecture focus was on how do we support very large clusters of coherent GPUs so we can connect these GPUs together with very high speed interconnects to create this large coherent cluster with high speed interconnects between each of the GPUs, they can share the memory that’s available to them. And so memory was really the constraint that we’re optimizing for. And so we create these very closely connected clusters to achieve that high performance computing infrastructure that allows us to create these amazing foundational models that we’re all familiar with today that started to shift as we put more of our focus into inference. So running workloads on top of those models. Early inference workloads were chatbots. They weren’t particularly complex and so those could be deployed into centralized infrastructure. And, and again, having those run on centralized infrastructure has got some values of you’ve got massive compute all in one location. You can often drive economies of scale there. You can drive efficiency in terms of placing workloads and optimizing usage across many users. We’re at the cusp now of the next wave of adoption where we’re moving into the rise of agents. So OpenClore, I’m trying to remember exactly how long ago it was. Six months ago, I think it kind of exploded onto the scene really kind of started the main wave of adoption around kind of agents. And what we’re seeing from our customers now is as I think about agents, these aren’t sort of one dimensional chat type applications, they’re multi stage, multi turn applications that it they’re calling tools, they’re looping multiple times around these kind of inference loops. They’re pulling in huge amounts of context to give the best answers possible. And physical geography is becoming a really important constraint in these architectures.
Swapnil Bhartiya: When enterprises try to force these new agenting workflow through that same legacy centralized infrastructure, what exactly breaks? Because often they don’t realize it, all they see is performance, it a higher bill or runaway token usage. But what is actually breaking underneath?
Jon Alexander: I think that the kind of simple answer is like, hey, the real world isn’t centralized data users, the devices, the APIs and all the decisions you’re making. They’re distributed. Like when we’re delivering a service in production, it doesn’t all sit in one data center. And like not everyone is physically close to that data center. And so if you think about it like what an agent really looks like, you’ve got a combination of the actual agent which runs on top of CPU infrastructure, it calls out to models that are running on GPUs. Those can often be multiple models. You can have different types of models calling, maybe it’s calling OpenAI, maybe it’s calling anthropic, maybe it’s calling open source models that are hosted on different GPU infrastructure elsewhere. So even the models aren’t all co located. But increasingly what we’re seeing is the agents are dominated by the tools that they’re calling. So the power of agents is around the memory system that it’s got, the access to the file system, the ability to execute code in some kind of a sandbox, generate output from the code that it’s running to call out to third parties. So to call APIs, to call MCPS, to call out to these external systems so it can interact with the real world. So this is the real power of an agent. And as it’s interacting with the real world, that’s not all in one location. And so this is the problem that we’re seeing is agents have to interact with the real world. That real world is distributed. Forcing all of that to run in one location means that you’re compounding latency. And these agentic systems, they run in a loop, they run multiple interaction or multiple times around the loop, and you’re compounding that latency on every loop. And so even adding 100 milliseconds of latency, if you’re looping 100 times, that’s 10 seconds of latency that you’ve added. Which might be okay for a chatbot, maybe a human willing to wait 10 seconds for an answer to come back. But for physical AI, for any type of real time system, 10 seconds is way too long. It’s a lifetime.
