By Most AI agents today send every request to the same centralized model, regardless of how simple or complex the task is. That static approach inflates latency, wastes compute on routine interactions, and eliminates any opportunity for intelligent, context-aware routing before a request reaches the core. The problem is not the models themselves. It is the absence of a smarter dispatch layer.
In this interview on TFiR, Dr. Robert Blumofe, Executive Vice President and Chief Technology Officer at Akamai, breaks down how edge infrastructure can serve as that dispatch layer, handling lightweight inference, intelligent routing decisions, and security functions at very low latency before traffic reaches centralized LLM infrastructure.
Guest: Dr. Robert Blumofe, Executive Vice President and Chief Technology Officer at Akamai
Show: TFiR
Here is what every platform engineer and AI infrastructure architect needs to know.
Technical Deep Dive
Q: Can edge infrastructure handle AI agent routing decisions instead of sending every request to a centralized LLM?
Dr. Robert Blumofe, Executive Vice President and Chief Technology Officer at Akamai, says yes, and the case is straightforward. AI agents perform many operations beyond invoking LLMs, and a significant portion of those operations, including routing decisions, can execute at the edge with very low latency. A lightweight model running at the edge can evaluate each incoming request and determine which model, which infrastructure, and which geographic location should handle it, before the request ever travels to a core data center.
“A fairly simple model running at the edge could probably make some good decisions about where the request should be routed, for the right model, the right infrastructure, in the right location.”
Dr. Robert Blumofe, Executive Vice President and Chief Technology Officer, Akamai
Q: Why is the current single-model-for-every-request approach a problem for AI agents?
Blumofe describes the current state as a fairly static world: organizations default to the same model for every interaction regardless of task complexity or latency requirements. There is no active decision about fit. That uniformity means simple requests consume the same compute and incur the same latency as complex ones, which is architecturally wasteful and increasingly unsustainable as agent workloads scale.
“Today we end up using the same model pretty much for every request that we make, every interaction. There’s no reason for that.”
Dr. Robert Blumofe, Executive Vice President and Chief Technology Officer, Akamai
Q: Which components of an AI agent workload are best suited to edge execution?
Blumofe frames AI agents as collections of many distinct components, and argues that many, though not all, of those components can run at the edge. These include intelligent routing, lightweight inference tasks, and security functions that require very low latency responses. Components that require heavier computation or access to larger models stay in more centralized infrastructure, connected to the edge via high-bandwidth, low-latency links.
“Many of the components, probably not all of them, but many of the components I think actually will run at the edge.”
Dr. Robert Blumofe, Executive Vice President and Chief Technology Officer, Akamai
Q: How does security fit into an edge-native AI agent architecture?
Blumofe explicitly includes security among the functions that belong at the edge. Security operations that require near-instant responses are a natural fit for edge execution, the same way they have been for traditional application delivery. In an AI agent context, this means security checks can run at low latency as part of the same edge layer that handles routing, without adding a separate round-trip to centralized infrastructure.
“Obviously there’s also security things that you do at very low latency at the edge.”
Dr. Robert Blumofe, Executive Vice President and Chief Technology Officer, Akamai
Q: Is the right architecture edge-only, cloud-only, or something in between for AI agent workloads?
Blumofe is explicit that neither extreme is correct. He describes the compute model as a continuum: some components run at the edge, some run in the core, and some run in between. The components that cannot run at the edge still benefit from the edge’s connectivity, reaching centralized infrastructure over low-latency, high-bandwidth paths. The result is a hybrid architecture where edge and core work together rather than compete.
“It’s a hybrid. There’s some things that are in the core, some things that are at the edge and things in between, and ultimately they work together to create a low latency, high bandwidth, high quality user experience.”
Dr. Robert Blumofe, Executive Vice President and Chief Technology Officer, Akamai
Resources & Documentation
- Akamai, distributed edge platform providing compute, security, and intelligent routing for AI and web workloads
***
👇 Click to Read Full Raw Transcript
Swapnil Bhartiya: Tying it back to your CDN roots. Could we eventually see smaller models cache it locally at the edge, similar to how we cache web content? How does that traditional edge model translate to the AI space?
Dr. Robert Blumofe: Yeah, I think the edge can be used in a lot of different ways. In the context of an AI application, you could do some of the computation at the edge. For example, when we talk about an agent, as I said, it does many, many different things, not just invoking LLMs. Some of those things could be done at the edge. Very, very low latency. Even some of the AI things you could be doing at the edge at low latency with, with relatively small models also that can include intelligent routing. You know, today again, it’s, we’re, we’re in a fairly static world in terms of what models we use, and we end up using the same model pretty much for every request that we make, every interaction. There’s no reason for that. And a fairly simple model running at the edge could probably make some good decisions about where the request should be routed. And you want to route for the right model, the right infrastructure in the right location. So those things can be considered when you make a rapid routing decision at the edge. Obviously there’s also security things that you do at very low latency at the edge. So again, ultimately, as you look at these AI applications, these AI agents, you break it down into many, many different components. And I think many of the components, probably not all of them, but many of the components I think actually will run at the edge and the ones that don’t will get very, very low bandwidth, low latency, high bandwidth connectivity from the edge to whatever more centralized infrastructure you need for that particular use case. So it’s a hybrid. You know, we oftentimes talk about the compute needs not being a one or the other, but, but as sort of a continuum, a hybrid. So there’s some things that are in the core, some things that are at the edge and things in between, and ultimately they work together to create a low latency, high bandwidth, high quality user experience.
