By AI inference placement is becoming one of the most consequential architectural decisions teams face. Centralizing inference for workloads that require sub-five-second responses, handle sensitive user data, or depend on real-time physical feedback introduces latency that directly translates to lost revenue, abandoned transactions, and security exposure. The industry does not yet have mature frameworks for making this decision systematically, and most teams are working it out under production pressure.
In this interview on TFiR, Ari Weil, VP Product Marketing at Akamai, walks through which workload categories genuinely require distributed AI inference, which do not, and what architectural and data placement questions teams need to be asking now.
Guest: Ari Weil, VP Product Marketing at Akamai
Show: TFiR
Here is what every platform engineer and AI infrastructure architect needs to know.
Technical Deep Dive
Q: Do all AI workloads benefit from running closer to users or data sources?
Ari Weil, VP Product Marketing at Akamai, is direct on this point: no. Pre-training, post-training, adding weights or embeddings, and model compression for memory efficiency are workloads that do not need to be federated or real-time. They can run centrally without latency penalty because the output is not delivered to a waiting user. The case for distribution only becomes compelling when inference is tied to an interactive, time-sensitive user experience or a physical process that cannot tolerate delay.
“If I were going to be training a model, pre-training or post-training, we don’t need those things to be federated. Those don’t have to be real time and responsive in this way.” — Ari Weil, VP Product Marketing, Akamai
Q: What is a concrete example of an inference workload that requires distributed or edge execution?
Weil uses an LLM-backed mortgage approval application as a primary example. When a user logs in, the system needs to retrieve user data, perform RAG-based credit and background checks, and return a decision, including loan amount and next steps, in under five seconds and certainly under thirty seconds. Each step in that pipeline must complete within that window or the user abandons the process. This makes centralized, high-latency inference architecturally unsuitable for the workload.
“All of it needs to come together in under five seconds, typically certainly under 30 seconds, and provide me with an answer that says you are or are not approved for this loan.” — Ari Weil, VP Product Marketing, Akamai
Q: How does the real estate industry illustrate the full complexity of distributed AI workload requirements?
Weil maps out the end-to-end workload of a real estate application to show that multiple distinct AI-driven workflows each carry different latency and distribution requirements. Listing discoverability now depends on zero-click search optimization and LLM scraper behavior, which is forcing immediate architectural rework on application and content layers. Scheduling a realtor walkthrough is a lightweight task with minimal AI value. Loan eligibility confirmation is high-value and latency-sensitive. Property transaction finalization involves multi-step approval automation. Each stream has a different profile.
“Zero-click searches and LLM scrapers are a phenomenon that already has a significant amount of right now architectural rework and application rework that companies are going through.” — Ari Weil, VP Product Marketing, Akamai
Q: Which industries are most exposed to the risk of centralized inference latency?
Weil identifies financial services as the clearest example, specifically insurance, warranties, and mortgages, where AI-driven decisions must be delivered quickly, securely, and reliably before the user loses confidence or worries their data has been compromised. The combination of high transaction value, regulatory sensitivity, and user trust requirements makes latency failure costly in ways that go beyond performance. Speed and security are not trade-offs in these workloads; both are required simultaneously.
“In the financial services industry, specifically around insurance and warranties and mortgages, we need to run quickly, securely and get an answer back to that user before they lose interest and walk away, or before they worry that maybe their information was compromised.” — Ari Weil, VP Product Marketing, Akamai
Q: How do anomaly detection and physical AI workloads fit the case for distributed inference?
Weil points to anomaly detection across a wide range of physical environments: autonomous vehicles, industrial robots, assembly lines, medical facilities, public sector operations, and military contexts. In each case, inference must happen at or near the source because the cost of a delayed response is physical, not just experiential. AR and VR devices performing real-time inference today in these environments are already demonstrating that edge-resident AI is not a future state; it is in production.
“There’s real-time inference happening through augmented reality and virtual reality goggles today that lend themselves to this.” — Ari Weil, VP Product Marketing, Akamai
Q: How does real-time multiplayer gaming illustrate distributed AI inference requirements?
Weil uses real-time multiplayer online gaming as a straightforward illustration of the latency requirement. Game worlds must be created, avatars and non-player characters must react and respond to live player input, and all of this must happen without perceptible delay. Any centralized inference bottleneck breaks the experience. Gaming is also a leading indicator for other sectors: the same real-time physical engagement pattern that drives gaming infrastructure investment applies to online sports broadcasting, news broadcasting, gambling, and e-gaming.
“You’re creating worlds, you’re having avatars and other characters in the game that are reacting and responding to people in real time. These types of things you keep hearing real time or where you have a physical engagement with the application, those are typically the things that would warrant distribution.” — Ari Weil, VP Product Marketing, Akamai
Q: What are the three core architectural questions that drive distributed AI infrastructure decisions?
Weil frames the decision-making process around three placement questions that companies are actively working through: where does the hardware go, where does the data go including the models themselves, and where will the application experience be interacted with. These are not independent decisions. The interaction point defines the latency budget, the latency budget constrains the hardware placement, and the hardware placement determines the data and model distribution strategy. Weil notes the industry is still in early stages of thinking through these systematically.
“Companies are starting to think really deeply about where does the hardware go, where does the data go, including the models, and then where is that application experience going to be interacted with. And that’s what’s driving the considerations.” — Ari Weil, VP Product Marketing, Akamai
Resources & Documentation
- Akamai, distributed cloud and edge platform for AI inference, security, and content delivery
***
👇 Click to Read Full Raw Transcript
Swapnil Bhartiya: Are there specific workloads or use cases that make the strongest case for running inference closer to users, devices or data sources? Or you feel that every use case, every workload is ideal if it is running closer?
Ari Weil: Oh no, definitely not. I mean if I were going to be training a model, pre training or post training of my model to do anything from, you know, adding weights to the model or embeddings that I might need later, if I’m, if I’m paring down a model that I might use, if I’m in a future workload and I want to make it a smaller overall model for storage reasons and what I can hold resident in memory, we don’t need those things to be federated. Those don’t have to be real time and responsive in this way. If I’ve already got a model and the use case is, hey, I have an LLM backed mortgage application or mortgage approval application, then I can think about when the user logs in, they’re going to provide me a certain amount of data. I might want to access my data about that user and maybe I’ll perform some rag activity to do a credit check or to do a background check on that person. But all of it needs to come together in under, you know, five seconds, typically certainly under 30 seconds and provide me with an answer that says you are or not approved for this loan. You are approved for this much and here are the next steps to actually capitalize on this loan. And if I think about that in a workload, like an end to end workload sort of perspective, imagine that I am a real estate company and I might have anything from my online listings. I need to make sure that all of my listings render completely correctly, consistently. But certainly that quickly piece needs to be top of mind for me. I need to ensure that in LLMs any zero click searches are actually having my page be cited more frequently than my competitors page so that people find me in the first place. Just finding that real estate listing already has a significant amount of right now architectural rework and application rework that companies are going through because of this phenomenon of zero click searches and LLM scrapers. Now that’s just again finding my listing. Next thing is maybe I want to understand can I set up a open house time, Can I schedule a walkthrough with the realtor? Maybe I want to find out what it might take for me to actually get approved for a loan on this property. Those are different work streams that I’m building into my application and each of those as well needs to be Real time. You would argue that scheduling something on the realtor’s calendar is a pretty lightweight use case. And you would be right. There’s not much of a appeal for AI in that sort of a use case. But confirming somebody’s eligibility and what level or what size of loan they could get is very much something that I can help with. And it’s very common in the financial services industry, specifically around insurance and warranties and mortgages, that we need to run quickly, securely and get an answer back to that user before they lose interest and walk away, or before they worry that maybe their information was compromised. So now I’ve got a listing that I’ve received, I’ve applied for a loan, maybe I’ve set up time for an open house. The next thing that I might consider is the actual set of transactions where I want to procure the home or I want to finalize that mortgage, I want to purchase the house, I want to go through other signatories. Those types of things typically as well will require some form of multi step approval process that is currently being automated by the people that are managing it. And so those types of things where you’re sitting there interacting in real time, where you’re expecting a response for something that is very high value and typically has a pretty good amount of risk associated with it, that’s a great example and a great use case for distributed AI because you are running inference and you want those answers to be fast, reliable and secure. If I think about some other ones that are related anytime that I might do anomaly detection, that could be in an autonomous vehicle, it could be in a robot or other physical AI device, it could be for augmented reality. If I’m doing work on a conveyor belt or an assembly line, I might be in the medical field, I may be in the public sector or even in a military sort of a facility. But there’s real time inference happening through augmented reality and virtual reality goggles today that lend themselves to this. And then we can think about even simpler things like real time multiplayer online gaming. You’re creating worlds, you’re having avatars and other characters in the game that are reacting and responding to people in real time. These types of things you keep hearing real time or where you have a physical engagement with the application, those are typically the things that would warrant distribution to maintain that low latency and that real time performance. But I think we’re just starting to understand this as an industry and even things like how we’ll create online, online sports, broadcasting, news broadcasting, gambling, e gaming things that really do reward latency and instantaneous responses are where we are seeing companies invest in this area and start to experiment with new ways of shuttling data across systems and starting to think really deeply about where does the hardware go, where does the data go, including the models, and then where is that application experience going to be interacted with? And that’s what’s driving the considerations.
