By AI inference is not the same workload as training, and the infrastructure decisions that work for one actively undermine the other. As AI agents move from experimental to production, the cost of placing inference in the wrong location compounds: latency degrades interaction quality, bandwidth constraints block multimodal delivery, and centralized architectures hit physical limits that more compute cannot solve.
In this interview on TFiR, Dr. Robert Blumofe, Executive Vice President and Chief Technology Officer at Akamai, breaks down why training and inference demand separate infrastructure strategies and what the shift toward multimodal AI agents means for where compute must live.
Guest: Dr. Robert Blumofe, Executive Vice President and Chief Technology Officer at Akamai
Show: TFiR
Here is what every platform engineer and AI infrastructure architect needs to know.
Technical Deep Dive
Q: Do AI training and inference genuinely require different infrastructure architectures?
Dr. Robert Blumofe, Executive Vice President and Chief Technology Officer at Akamai, confirms that training and inference have fundamentally different proximity requirements that drive different infrastructure decisions. Training workloads require proximity to the training data set, which is typically large and stored in a centralized storage cluster. Inference workloads require proximity to the things the AI is interacting with, primarily users, which are distributed across large geographies.
“It’s helpful to think about what is the important affinity, what does the infrastructure need in terms of proximity.” — Dr. Robert Blumofe, Executive Vice President and Chief Technology Officer, Akamai
Q: Why does AI training work well in a centralized, GPU-dense cluster?
Blumofe explains that training data sets are large and generally stored in a single centralized storage cluster rather than distributed across many locations. Because training computation is also very GPU-dense, co-locating a dense GPU cluster with the training data set in one centralized location is the architecture that makes sense. Moving data across large distances to distributed compute nodes would introduce unnecessary complexity and latency at the data ingestion layer.
“A dense GPU cluster centralized where the training data set is — that makes a whole lot of sense.” — Dr. Robert Blumofe, Executive Vice President and Chief Technology Officer, Akamai
Q: What determines where AI inference infrastructure should be placed?
For inference, Blumofe states the key affinity is proximity to the entities the AI agent is interacting with. Users are a primary interaction point for agents, and users are distributed across countries, continents, and globally. Placing inference compute thousands of miles away from users in a single centralized location directly conflicts with the latency and interaction quality requirements of real-time agent conversations.
“It makes no sense really for the agent that you and I are interacting with to be thousands of miles away, centralized in a single location.” — Dr. Robert Blumofe, Executive Vice President and Chief Technology Officer, Akamai
Q: How are AI agent interactions changing beyond text and voice?
Blumofe draws a parallel to how the web evolved from simple pages to high-definition video as bandwidth and infrastructure matured. He argues the same evolution is underway with AI agents: interactions will extend beyond text and voice to include video and dynamically updating images, modalities that were not available before AI. These high-bandwidth interaction modes deliver compelling value but place significant demands on the delivery infrastructure.
“A good agent is going to show us video, is going to show us images, and it’s going to be dynamically updating the video and dynamically updating the images.” — Dr. Robert Blumofe, Executive Vice President and Chief Technology Officer, Akamai
Q: Why can’t centralized infrastructure scale to meet multimodal AI agent demand?
Blumofe is direct: there is no way to deliver high-bandwidth, multimodal AI agent interactions at scale through a brute-force buildout of centralized infrastructure. As agents become ubiquitous and interactions shift to video and dynamic images, the physics of distance and the bandwidth requirements of those modalities make centralized delivery architecturally insufficient, regardless of how much compute is added in a single location.
“There’s just no way to do that through a brute force build out in centralized infrastructure.” — Dr. Robert Blumofe, Executive Vice President and Chief Technology Officer, Akamai
Resources & Documentation
- Akamai, distributed cloud platform providing edge compute and delivery infrastructure for AI inference at scale
***
👇 Click to Read Full Raw Transcript
Swapnil Bhartiya: Let’s look at training versus inference. Most companies care far more about inference to actually deliver AI to their users. Do these two phases genuinely demand completely different infrastructure architectures?
Dr. Robert Blumofe: Yeah, it’s a great point and a great way to sort of distinguish these use cases. And I think it’s helpful to think about what is the important affinity, what does the infrastructure need in terms of proximity? And arguably in the case of training, the key affinity, the key proximity requirements is the data set, the training data set. And most training data sets are fairly large and they’re generally stored in, in some large storage cluster that’s going to be fairly centralized. You typically wouldn’t have your training data set distributed around a large number of locations. It’s going to be fairly centralized. So it makes sense to do the training where the training data set is. It’s also the case that if you look at the actual computation, you know, it’s very GPU dense. So a dense GPU cluster centralized where the training data set is, that makes a whole lot of sense when you move into inference. Well, what’s the affinity? What does it need to be near? Well, it needs to be near the things that it’s interacting with. And, and there’s a lot of things that, that AI applications, AI agents have to interact with. But obviously one of the important users of course, is the people. Us. You know, we are going to use agents to get things done for us. We’re going to engage in conversations with these agents to help specify what it is we want done at results, to review results, provide feedback. It’s going to be very conversational. So the affinity of an agent, I mean, we can get into this a little bit more in a little bit because there’s a lot of different affinities, but one of them clearly is to the users. And users are typically distributed over a fairly large swath of geography, whether it’s a country or a continent or the entire world. So it makes no sense really for the agent that you and I are interacting with to be thousands of miles away, centralized in a single location. And relative also to what I was mentioning earlier about the nature of the, of the interaction changing with the web, where we went to high definition video and things like that. The same thing is the case with agents. You know, you don’t want to think of an agent interaction as being simply text or even voice. A good agent is going to show us video, is going to show us images, and it’s going to be dynamically updating the video and dynamically updating the images. These are capabilities that you don’t get outside of AI. And it’s one of the great, great benefits of using an AI agent is that you have all these modalities available, video, images, interaction that isn’t available in with other technologies. So, you know, as we see agents become more ubiquitous, I think we’ll see these high bandwidth forms of interaction really take, take hold because they really deliver value and they deliver something compelling and interesting. And there’s just no way to do that through a brute force build out in centralized infrastructure.
