By GPU saturation in centralized cloud regions does not produce linear latency degradation. As utilization climbs, queuing delays grow exponentially and batching decisions that performed acceptably at low load begin adding hundreds of milliseconds to time to first token. Most teams only discover this failure mode after a production workload is already dependent on the architecture they chose.
In this interview on TFiR, Ari Weil, VP Product Marketing at Akamai, breaks down why single-region inference deployments fail under production load, how distributed cloud infrastructure addresses the proximity gap, and what enterprises need to plan now to align their cloud vendor roadmaps with inference workload requirements through 2027 and beyond.
Guest: Ari Weil, VP Product Marketing at Akamai
Show: TFiR
Here is what every platform engineer and AI infrastructure architect needs to know.
Technical Deep Dive
Q: Why does centralized cloud architecture break down for AI inference workloads?
Ari Weil, VP Product Marketing at Akamai, points to a fundamental mismatch between how centralized cloud was designed and what production inference workloads actually require. Survey data shows 60% of practitioners identify proximity to the user as critical, yet 46% of inference workloads still run in a single centralized cloud region. Weil notes that this gap reflects organizations learning architectural lessons too late, typically after a workload has already been deployed and the business impact of latency has become measurable.
“The centralized cloud was built for an era when compute was something that you went to. In the agentic era, we need companies that are going to bring the compute closer to you.” — Ari Weil, VP Product Marketing, Akamai
Q: How does workload maturity change the requirement for inference proximity?
Weil draws a clear line between experimentation and production. In early experimentation, only about 30% of teams identify proximity as critical. Once a workload becomes a core business function, that figure rises to 77%. This pattern shows that proximity requirements are not apparent during proof-of-concept phases but become unavoidable at production scale, and teams that do not build proximity into the architecture at the proof-of-concept stage face expensive redesigns later.
“When it’s your core business workload, 77% of companies said that proximity is critical. People are starting to figure out, unfortunately, too late in the process.” — Ari Weil, VP Product Marketing, Akamai
Q: Where are the businesses driving distributed inference deployment actually located?
Weil clarifies a common assumption: while digital sovereignty discussions in Europe draw attention, the majority of businesses deploying distributed inference workloads are located in North America, specifically the United States, or in Asia Pacific regions including China, India, and Japan. Weil identifies Asia Pacific as a region showing particularly aggressive scaling of inference workloads, making it a leading indicator of where distributed inference architecture is being adopted fastest.
“The majority of those businesses are located in North America, specifically the United States, or in places in Asia Pacific, primarily China and India and Japan, where we see a lot more aggressive scaling of inference workloads.” — Ari Weil, VP Product Marketing, Akamai
Q: Why does GPU saturation cause exponential latency growth in centralized inference deployments?
Weil explains that centralized inference architectures assume latency will scale roughly linearly with load, but that assumption does not hold. As GPU utilization climbs toward saturation, queuing delays grow exponentially. Batching decisions that performed acceptably at low load begin adding tens or hundreds of milliseconds to time to first token, and if the GPU cluster was not sized correctly from the start, requests may be routed to GPUs that are not physically close enough to one another, compounding the delay further.
“Your queuing delays start to get exponentially larger and batching decisions that used to work at low loads start adding tens or hundreds of milliseconds to that time to first token.” — Ari Weil, VP Product Marketing, Akamai
Q: How do geographic distribution and GPU queuing interact to compound inference latency?
Weil describes two distinct latency dimensions that compound in centralized deployments: the round-trip time from user to data center, and the queuing time to access saturated GPU resources within that data center. When both are present simultaneously, the combined latency impact becomes severe. Weil frames this as load revealing the architecture, meaning the choice to use a centralized single-region deployment is not visible as a problem until the workload grows enough to expose it.
“Your load is going to reveal the sort of architecture that you chose.” — Ari Weil, VP Product Marketing, Akamai
Q: How will AI infrastructure split between centralized training and distributed inference?
Weil describes an emerging architectural split driven by the different hardware and facility requirements of training versus inference. Hyperscalers such as Oracle Cloud Infrastructure are doubling down on centralized AI factories with heavyweight GPUs including H100s, H200s, and L40s optimized for batch training workloads. Inference workloads, by contrast, favor distributed, air-cooled, smaller data center footprints because they do not require the same power, cooling, or cabinet density, which is precisely the infrastructure profile Akamai has built over 28 years.
“I’ve spent the last 28 years building out a very broadly geographically distributed network of relatively small data centers that I can now use primarily for inference workloads because they lend themselves to it.” — Ari Weil, VP Product Marketing, Akamai
Q: What does tokenomics mean for how inference workloads must be aligned to hardware specifications?
Weil points to tokenomics as a framework emerging to give enterprises clearer insight into how a specific workload needs to align to a specific machine specification, including the amount of available RAM and the balance of GPU versus CPU processing. The core insight is that scaling is not always linear, and understanding tokenomics helps teams predict when a given hardware configuration will saturate and what the latency consequences of that saturation will be before it occurs in production.
“We need to better understand that scaling isn’t always linear.” — Ari Weil, VP Product Marketing, Akamai
Q: How should enterprises approach vendor selection and roadmap alignment for distributed inference?
Weil recommends that enterprises actively diversify their cloud vendor pipeline rather than consolidating on a single provider, and evaluate vendors based on where they are going rather than where they are today. The key criteria are: which providers have the network and geographic reach to deploy specialized inference hardware where the workload actually needs to run, and whether a given enterprise is large enough in that provider’s roadmap to influence delivery timelines. Weil frames early partnership and roadmap alignment as the new competitive differentiator in AI infrastructure through 2027 and into 2030.
“You start to really think about your roadmap and your partners based on where they’re going versus where they are, because you need the ability to steer them in the direction that is going to be consistent with the workload and the architecture that you need.” — Ari Weil, VP Product Marketing, Akamai
Q: What will the cloud ecosystem look like by 2030 as distributed inference scales?
Weil predicts the competitive dynamic in cloud will shift away from raw power, cooling capacity, and centralized data center scale, which dominated from roughly 2023 onward, toward specialization, network reach, and the ability to deploy the right hardware type in the right geographic location. He expects new specialized providers to emerge from the current set by 2030, having evolved specifically to serve the low-latency distributed inference requirements that are now becoming clear. The arms race, in Weil’s framing, is no longer about who has the largest GPU cluster but about who has the right infrastructure in the right place.
“It’s going to be what is the specialization you need, who’s got the network and the ability to deploy that specialization based on where you need it.” — Ari Weil, VP Product Marketing, Akamai
Resources & Documentation
- Akamai Cloud Computing, distributed cloud infrastructure optimized for inference workloads across globally distributed, air-cooled data centers
***
👇 Click to Read Full Raw Transcript
Swapnil Bhartiya: The interesting thing is that with all this AI, whether we are looking at inferencing or, I mean that’s what we focus on. We are not training models, companies. Cloud is going to play, not going to play. It is actually the foundation of it. Can you talk about where does the traditional centralized cloud model start to break down for inference heavy workloads? Where we also need to look at decentralized approach. Also we hear a lot about new clouds these days, especially in Europe, because of also we can also throw the whole AI sovereignty, digital sovereignty that is going on in Europe a lot, but it will also become global phenomena.
Ari Weil: Yeah, well, I think, look, going back to the survey results that we saw, 60% of the practitioners that we surveyed said that proximity to the user is critical. But we also saw that in 46% of cases, their inference workloads are still running in a single centralized cloud region. And I think that gap is where we are starting to see people really learning some tough lessons about how to architect across NEO clouds, hyperscalers, alternative clouds, you know, and the rest of the folks who are competing in the ecosystem. And I think this is also an area where a lot of the tier one analysts are trying to, to really catch up and adjust their taxonomies. Because if we look at data by maturity stage, then people in their early experimentation, only about 30% really identify proximity as critical. But when it’s your core business workload, 77% of companies said that that proximity is critical. And I think that really speaks to when you start figuring out the business impact of the workload that you’re building and where you actually rely on that sort of round trip time. That’s where people are starting to figure out, unfortunately, too late in the process. I mentioned before, when you do your first proof of concept or proof of value or whatever you consider it, that’s when you need to start building in the infrastructure and the architectural relevance of where you are serving that pocket. Because if we look at the people who are running single region deployments, And I said 77% of those workloads, they realized that proximity was important. Under 14% of those workloads are deployed in a centralized region or a single region. And I think that just shows you when you get down to over 85% of workloads being distributed, when they realize that latency matters and proximity is an important part of that latency equation, then you have a much clearer picture on trajectory. And, and then you start asking yourself where are those businesses located? If you thought that they were located in areas where for example, sovereignty and privacy are top bill items like across emea, that would be a logical conclusion to draw, but it’s not the case. The case is that the majority of those businesses are located in North America, specifically the United States, or they’re in places in Asia Pacific, primarily in places like China and India and Japan, where we see a lot more scaling and more aggressive scaling of these inference workload. And so, so I think for us, what we’re realizing that the centralized cloud was built for an era when compute was something that you went to, you had to go where the people had cards, horsepower storage, et cetera. In the agentic era, we need something different. We need companies that are going to bring the compute closer to you. And more and more, not just users, but also people who are architecting their next applications are going to start being a lot more selective about picking the right card and the right infrastructure for their workloads and putting that in the right place for the users that they have to deploy to. That’s a completely different set of calculus than you used to apply in a centralized cloud region, because now it’s not so much about just reserved instance capacity and committed revenue. Now it’s really thinking about where and when do I have to scale? Which is going to move us, I think to a lot more of a just in time sort of deployment architecture than we’ve had in the last 20 years.
Swapnil Bhartiya: Do you feel that as of course, AI goes more and more into production, it is already in production. Do you see that AI infrastructure will become more hybrid with training will be centralized, but inference will be more distributed. As you also mentioned that they do want it to be closer to user, but it’s still centralized, far from them. So how do you see training versus inference where they will run?
Ari Weil: I think the, the interesting thing is where we look at tokenomics and some of the companies that are really starting to push that idea forward and to give us a lot more insight into how a workload needs to align to a certain spec of machine, to a certain amount of, for example, RAM that’s available to GPU and CPU processing. The thing that’s happening is we need to better understand that scaling isn’t always linear. So centralized inference architectures always are going to assume that latency will roughly stay the same as the load increases and that doesn’t actually exist. That doesn’t occur as the GPU utilization is going to climb and you start thinking about saturating the available GPUs in a given location, then your queuing delays Start to get exponentially larger and batching decisions that used to work at low loads. Start adding tens or hundreds of milliseconds to that time to first token because your response time is just going to start shooting off the more that you have in queue and the more that you’re going to be sensitive to either adding additional GPUs to the footprint that you have, or having GPUs that might not be close enough to one another because you didn’t plan for the appropriate size of GPU cluster and you’re now routing away from where some of your requests are coming. So you start thinking about, I’ve got a scaling dimension for how much I can scale up my GPUs. Then I have a challenge around geographic dimensions. So you think about round trip times now being exacerbated by queuing times where I have to get access to resources in my centralized deployments, and you start realizing that your load is going to reveal the sort of an architecture that you chose. And so I think if we put all of this stuff together, the really interesting challenge is going to be how different cloud providers. Whether you’re a NEO cloud that’s primarily focusing on a lot of the tensor core architectures, the GPUs that you need to drive sort of AI workloads. But classically it would have been more of the training and less of the inference workloads. And then we see everybody diversifying and starting to add more of the cards that you need for inference. And then we start seeing these sort of hybrid or alternative types of clouds pop up and you can’t use either word because they already mean something in the cloud space. But you have people that don’t have as many locations. They might not have as much hardware, but they’re very specialized for a given domain. Now you’ve got this really challenging architectural problem of how do I start planning my capacity based on available providers and help my providers understand how they should be building out their next infrastructure buys. Because if I’m a company like Akamai, I’ve spent the last 28 years building out a very broadly geographically distributed network of relatively small data centers that I can now use primarily for inference workloads because they lend themselves to it. They’re an air cooled architecture. I don’t need the same size cabinets, I don’t need as much power, I don’t need as much cooling. And so I will be able to build out a lot of globally distributed inference architecture if I’m a hyperscaler and somebody like an oci, for example, or Oracle. They are now investing and really doubling and tripling down on a centralized infrastructure where they want to build out AI factories with very heavyweight GPUs like H1 hundreds, H2 hundreds, L40s and things of that nature that are specifically tuned for batch type of workloads. Well, how do I start going from my centralized batch to my distributed inference when I’m a business that’s trying to scale my AI? And the answer is you start to really diversify your vendor pipeline and think about who’s going to be providing you with access to compute and GPU and maybe in the future TPU architectures that you’re going to need for your workload, where those workloads are being deployed. And then you start to really think about your roadmap and your partners based on where they’re going versus where they are, because you need the ability to steer them in the direction that is going to be consistent with the workload and the architecture that you need. And at this point, there are some very large and very strategic companies that are steering this path for a lot of us out in the marketplace because they have the capital, the forethought and the maturity to already be steering people in a given direction. But I think that is going to be the new arms race. It’s not going to be centralized data centers, it’s not going to be raw power or cooling resources that we’ve been seeing for, you know, call it the last three years very intensively now. It’s going to be what is the specialization you need, who’s got the network and the ability to deploy that specialization based on where you need it? And how can you partner with them early enough and be meaningful enough in their roadmaps to ensure that their delivery roadmap aligns to yours? That is a really interesting challenge that we’re currently facing in 2026, going into 2027 and beyond. But I think by the time we get to 2030, it will have completely reshaped what the cloud ecosystem looks like with some new specialized providers really emerging out of the current set, because they will have evolved to treat these specialized needs of distributed low latency inference workloads.
