By GPU capacity has become the default benchmark for AI infrastructure investment. But in agentic systems, where models don’t just generate answers but take actions, tool calling, file I/O, and code execution consume the majority of wall-clock time. Optimizing only for token generation throughput leaves the largest share of latency untouched.
In this interview on TFiR, Jon Alexander, SVP of Product, Cloud Technology Group at Akamai, breaks down where agentic AI execution time actually goes and what infrastructure teams need to coordinate to improve real-world performance.
Guest: Jon Alexander, SVP of Product, Cloud Technology Group at Akamai
Show: TFiR
Here is what every platform engineer and AI infrastructure team needs to know.
Technical Deep Dive
Q: Is GPU capacity the right metric for optimizing agentic AI performance?
Jon Alexander, SVP of Product, Cloud Technology Group at Akamai, argues that GPU capacity is necessary but not sufficient. Based on patterns seen across customer deployments of real-world agentic applications, Alexander identifies GPUs as one of the most expensive and constrained parts of the architecture, making effective utilization critical. However, the data from live deployments shows that a large portion of total execution time does not occur on GPUs at all.
“GPUs are incredibly important. They’re one of the most expensive and constrained parts of the architecture. So capacity there is limited. You want to make sure you’re using it effectively.”
Jon Alexander, SVP of Product, Cloud Technology Group, Akamai
Q: Where does agentic AI execution time actually go if not on the GPU?
Alexander points to tool calling as the dominant consumer of execution time in agentic workflows. When an agent calls out to third-party systems, writes files to the local file system, or executes code, that work runs on CPUs. Across real-world deployments, this CPU-bound work can account for up to 90% of overall execution duration, leaving GPU token generation as a minority share of total latency.
“Up to 90% of the overall execution time isn’t on the GPU. It’s actually running on CPUs where it’s doing this other work.”
Jon Alexander, SVP of Product, Cloud Technology Group, Akamai
Q: What does this mean for how teams should architect agentic AI infrastructure?
Alexander frames end-to-end latency in agentic systems as a coordination problem, not a raw compute problem. Orchestrating work across CPUs and GPUs, placing infrastructure in the right locations, and ensuring reliable connectivity to external systems are all variables that directly control performance in production. Teams that focus exclusively on GPU throughput will hit diminishing returns while leaving the larger latency drivers unaddressed.
“Orchestrating that and coordinating between the CPUs and the GPUs, strategically making sure that you’re leveraging that infrastructure in the right locations, making sure you’ve got connectivity to external systems, all of that is as important, if not more important, for improving the performance in these real world scenarios.”
Jon Alexander, SVP of Product, Cloud Technology Group, Akamai
Resources & Documentation
- Akamai Cloud Technology Group, distributed cloud infrastructure optimized for agentic AI and edge compute workloads
***
👇 Click to Read Full Raw Transcript
Swapnil Bhartiya: No matter who you talk to, the conversation always comes back to GPU capacity and token generation. Is that focus right or is it incomplete? What are organizations actually missing and what should they be focused on? As we shift from generating answers to taking actions,
Jon Alexander: this is one of the kind of big items that we’ve identified as we’re seeing customers deploying real world agentic applications. GPUs are incredibly important like that. They’re one of the most expensive and constrained parts of the architecture. So capacity there is limited. You want to make sure you’re using it effectively. But the way that we’re seeing these agentic applications built, a large portion of the execution duration isn’t waiting for the GPU to generate tokens to generate the response. And it’s actually in the tool calling. So the agent is calling out to third party systems, it’s writing files to the local file system, it’s executing code. So it’s doing all of this other work. And that can be up to 90% of the overall execution time isn’t on the GPU. It’s actually running on CPUs where it’s doing this other work. And so you definitely see a lot of benefit from the GPUs being fast. That can be a bottleneck. But increasingly we’re seeing that the end to end latency is being controlled by all these other parts of the agentic system. And so orchestrating that and coordinating between the CPUs and the GPUs strategically making sure that you’re leveraging that infrastructure in the right locations, making sure you’ve got connectivity to external systems, all of that is as important, if not more important, for improving the performance in these real world scenarios.
