By Teams deploying agentic AI applications are tuning GPU throughput and token generation speed while the real bottleneck sits elsewhere. In distributed agentic workloads, tool calling, external API requests, and real-time data retrieval can account for up to 90% of total task execution time. No GPU upgrade resolves that. Infrastructure placement and orchestration do.
In this interview on TFiR, Jon Alexander, SVP of Product for the Cloud Technology Group at Akamai, covers how the compute continuum and Akamai’s AI Grid Orchestrator address end-to-end agentic latency by separating and optimally placing CPU and GPU workloads across distributed infrastructure.
Guest: Jon Alexander, SVP of Product for the Cloud Technology Group at Akamai
Show: TFiR
Here is what every platform engineer and AI infrastructure architect needs to know.
Technical Deep Dive
Q: What performance metrics actually matter for real-world AI agent applications beyond GPU throughput?
Jon Alexander, SVP of Product for the Cloud Technology Group at Akamai, explains that time to first token and tokens-per-second remain important GPU metrics, but they capture only a fraction of agentic workload performance. In agentic applications, up to 90% of total task execution time occurs outside the GPU entirely, in the tool calling layer where agents make requests to third-party systems, external APIs, and real-time data sources. Optimizing GPU performance without addressing that layer leaves the majority of end-to-end latency untouched.
“Often what we’re seeing is up to 90% of the overall task execution isn’t on the GPU. The majority of that time is actually in the tool calling portion of the agent.” — Jon Alexander, SVP of Product for the Cloud Technology Group, Akamai
Q: Why does compute location affect AI agent latency even when running the fastest available GPU?
Alexander points out that even the most capable GPU model generates slow, unsatisfactory results if the surrounding infrastructure is poorly positioned. Poor connectivity to the tools agents call, to storage systems, and to real-time data sources introduces latency that no GPU upgrade can offset. The end-to-end system architecture, not the GPU specification, determines whether an agentic application meets user expectations.
“You can have the most powerful fastest model running on the latest GPU. But if your compute is in the wrong location, if you don’t have good connectivity out to the tools, your system is still going to be slow.” — Jon Alexander, SVP of Product for the Cloud Technology Group, Akamai
Q: What is Akamai’s compute continuum and how does it apply to AI workloads?
Alexander describes the compute continuum as Akamai’s architectural approach to spanning centralized data centers through to the network edge, treating the full range of compute locations as a single addressable infrastructure for workload placement. For AI specifically, this means large models requiring clusters of powerful GPUs can remain in centralized locations where utilization is manageable, while CPU-bound agent components, sandboxes, and containers handling individual user sessions can be placed closer to end users to reduce latency and cost.
“As we move from centralized token factories to real world systems that can deploy applications for real time token generation, having the ability to have that in a grid distributed infrastructure is important, but that needs orchestration.” — Jon Alexander, SVP of Product for the Cloud Technology Group, Akamai
Q: What is Akamai’s AI Grid Orchestrator and what problem does it solve?
The AI Grid Orchestrator is Akamai’s solution for coordinating CPU and GPU workloads across a distributed infrastructure footprint. Alexander notes it is designed to align with Nvidia’s AI grid reference architecture and addresses the scheduling complexity that arises when different components of an agentic application need to run in different physical locations. The orchestrator places each workload component where it produces the best balance of latency and infrastructure utilization, then coordinates execution across those distributed locations.
“Key requirement for the orchestrator is to coordinate between the CPU and the GPU and schedule the execution effectively across those different components of the infrastructure.” — Jon Alexander, SVP of Product for the Cloud Technology Group, Akamai
Q: Why does separating CPU and GPU workloads benefit distributed agentic AI systems?
Alexander explains that highly distributed GPU clusters are difficult to maintain at effective utilization, making centralized placement the practical choice for large model inference. CPU workloads, by contrast, are session-scoped: each sandbox or container is isolated to an individual user or session, making edge placement viable and beneficial for latency. Separating the two allows each to be placed optimally rather than forcing both to share suboptimal infrastructure to stay co-located.
“The separation between CPU and GPU is often beneficial in these types of distributed systems. We can place those different components of the application in the right physical location to drive end to end latency down.” — Jon Alexander, SVP of Product for the Cloud Technology Group, Akamai
Resources & Documentation
- Akamai Cloud Computing, Akamai’s distributed cloud platform spanning data centers and edge locations for AI and application workloads
- Nvidia AI Grid Reference Architecture, Nvidia’s reference design for distributed GPU infrastructure that Akamai’s AI Grid Orchestrator aligns with
***
👇 Click to Read Full Raw Transcript
Swapnil Bhartiya: In traditional cloud we talk about CPUs, storage, networking, memory, but when it comes to AI, we mostly talk about GPU capacity and token generation speed. What metrics are missing from that conversation? The ones that actually matter for real world AI agent performance?
Jon Alexander: Yeah, no, this is kind of a key optimization that we’ve been looking at again is as we’ve been working with customers, is that they’re building these agents. The GPU is really important. We’ve been deploying a lot of the Blackwell GPUs and the speed of those GPUs. So time to first token, the throughput, the number of tokens they can generate per second, those are incredibly important metrics that we’re seeing that customers are optimizing for. But for these applications, in a gentic application, often what we’re seeing is up to 90% of the overall task execution isn’t on the GPU, so isn’t kind of generating tokens. The majority of that time is actually in the tool calling portion of the agent. So calling out to these third party system calling external APIs, pulling data into, pulling in context, pulling in real time information that’s needed to generate the answers. This is what’s actually generating the majority of the end to end latency for the task to be successful. And so optimizing for the end to end system is what’s important. So GPUs are a really important part, but often what we’re seeing is there’s a really big portion of the end to end time that’s not on the GPU at all. So you can have the most powerful fastest model running on the latest gpu. But if your compute is in the wrong location, if you don’t have good connectivity out to the tools, if you’re not running with good connectivity to storage and other data systems that you need, your system is still going to be slow and isn’t going to meet the expectations of the users.
Swapnil Bhartiya: Akamai has been talking about a compute continuum for a while now. What does that actually look like in practice? Spanning from centralized data centers all the way to the network edge. When we talk about AI, this is
Jon Alexander: a big optimization problem. So something that we’ve been working on is what we call our AI grid orchestrator. So Nvidia has talked a lot about their kind of AI grid reference architecture. So as we move from kind of centralized token factories to kind of these real world systems that can deploy applications for kind of real time token generation, having the ability to have that in a grid distributed infrastructure is important, but that needs orchestration. So you’ve got all of these different locations where you can run workloads. And what we find is there are certainly very large models that need to be deployed on powerful GPUs, often clusters of GPUs. Makes sense to have that in more centralized locations. So you don’t want to have those deployed in hundreds of locations at very hard to maintain utilization of highly distributed clusters of GPUs like that. But then what we see is for the cpu, where the agent is running, where we’re deploying sandboxes, where we’re spinning up containers to kind of run code execution, that’s something that is isolating sandbox to an individual user, an individual session. And it makes sense as we look to optimize end to end performance, manage cost for that to be close to the end user. That’s where what we’re seeing is the separation between CPU and GPU is often beneficial in these types of distributed systems. We can place those different components of the application in the right physical location to drive end to end latency down and then also make sure we’re driving effective utilization of that infrastructure. And then key requirement for the orchestrator is to coordinate between the CPU and the GPU and schedule the execution effectively across those different components of the infrastructure.
