By Real-time AI workloads, including voice agents, fraud detection, and speech-to-text, break down when inference is concentrated in centralized cloud regions. Teams that replicate models across locations to close the latency gap pay twice for the same answer without always recognizing it as an architectural cost decision. The underlying problem is that model placement is treated as an operational afterthought rather than a design-time choice with direct financial consequences.
In this interview on TFiR, Ari Weil, VP Product Marketing at Akamai, walks through how practitioners should evaluate model size, placement strategy, orchestration requirements, and the cost trade-offs between copying a model for proximity versus routing requests intelligently to a single authoritative instance.
Guest: Ari Weil, VP Product Marketing at Akamai
Show: TFiR
Here is what every platform engineer and AI infrastructure architect needs to know.
Technical Deep Dive
Q: How do standard SaaS cost metrics like ROI, TCO, and CAC apply to AI workloads?
Ari Weil, VP Product Marketing at Akamai, explains that the same financial frameworks used in SaaS, return on investment, total cost of ownership, and customer acquisition cost, apply to AI but behave differently because AI introduces new variables around model size, placement, and data movement that do not exist in conventional cloud workloads. In traditional cloud environments, scaling means deploying additional compute or microservices to more locations, which addresses both resilience and latency in a relatively predictable cost model. With AI, the question of whether to replicate the model itself, not just compute, introduces a cost layer that practitioners familiar with SaaS TCO calculations have not had to account for before.
“Those are three numbers, ROI, TCO, and CAC, that many people in SaaS environments are very used to looking at all the time. But if we think about where AI is a little bit different in this equation…” — Ari Weil, VP Product Marketing, Akamai
Q: Why can’t you simply scale AI workloads the same way you scale microservices in the cloud?
With conventional microservices, deploying to additional environments increases availability, resilience, and throughput without introducing major data consistency concerns. With AI, scaling out means deciding whether to replicate the model itself across locations, and that decision carries cost, data movement, and model freshness implications that have no direct parallel in microservices architecture. Weil notes that many developers have not yet worked through whether it is economically viable or architecturally correct to deploy a model in multiple locations given those constraints.
“A lot of companies don’t have the experience, a lot of developers haven’t yet figured out: can I afford to, or does it make sense to deploy my model in multiple locations?” — Ari Weil, VP Product Marketing, Akamai
Q: What is the cost difference between deploying a fine-tuned model versus a large frontier model?
Fine-tuned and domain-specific models are typically smaller than frontier large language models because they are trained only on the data relevant to a specific use case or industry, which means they do not require the same breadth of training data or parameter count. For application developers building domain-specific tools where accuracy, low hallucination rates, and fast response times matter, a fine-tuned model is likely a better fit than a frontier model and carries lower infrastructure cost per inference. The smaller footprint also affects placement decisions, since a smaller model is more practically deployable across multiple locations if proximity is required.
“If I am a domain-specific application developer and I need my app to give me quick responses and not hallucinate, I might be looking at a fine-tuned model that will typically be smaller than these frontier large language models.” — Ari Weil, VP Product Marketing, Akamai
Q: When does copying an AI model to multiple locations make sense versus when does it create unnecessary cost?
Replicating a model for proximity is justifiable when the model changes infrequently, the latency budget is strict, and the cost of the duplication is less than the cost of the performance degradation it prevents. The problem arises when teams replicate a model purely to overcome a latency barrier without accounting for the fact that they are now paying twice for the same answer, and without evaluating whether intelligent request routing to a single model instance would achieve the same result at lower cost. Weil frames this as a design-time and architecture-time decision that must be made deliberately, not reactively.
“If I could access one model from multiple locations, but I’m copying that model to overcome a latency barrier, then I’m incurring additional costs simply to overcome latency.” — Ari Weil, VP Product Marketing, Akamai
Q: Which AI workload types require low latency and cannot tolerate centralized inference?
Weil identifies voice agents, speech-to-text, video-to-text, and real-time fraud or anomaly detection as workload types where latency is not a quality-of-service concern but a functional requirement. These use cases involve users who are already conditioned to expect real-time responses, meaning any perceptible delay is a failure mode, not just a performance issue. Centralizing these workloads in a single region works well for users close to that region but degrades progressively for geographically distant users, and the compensating measures teams typically implement, such as model replication, add cost without addressing the root architectural problem.
“If you need anomaly detection or fraud detection, or to provide somebody with a real-time query where they’ve already been conditioned to anticipate a real-time response, then you need to be thinking about orchestration capability to decompose a request and go to where the right information is.” — Ari Weil, VP Product Marketing, Akamai
Q: When does the centralized cloud thesis actually work for AI inference?
Centralized inference is appropriate when the workload does not sit in the critical path of a user-facing request, when usage is sparse or infrequent enough that geographic distance from the region does not produce meaningful latency impact, or when the request does not require a real-time response. Weil is explicit that the centralized model is not wrong in general, it is wrong when applied to workloads that have real-time requirements or that serve users distributed far from the central region. The cost and performance penalty for misapplying the centralized model grows proportionally with the interactivity and geographic spread of the user base.
“The centralized cloud thesis makes sense if you have centralized functions that are sparsely or sparingly used, that are not directly in the request-response timing for an end user request, or if that request doesn’t require real time.” — Ari Weil, VP Product Marketing, Akamai
Q: What is the role of orchestration in distributed AI inference architecture?
Orchestration is the layer that allows a system to decompose an incoming request and route it to the location where the relevant model, data, or code is managed, rather than requiring every piece of infrastructure to be replicated everywhere. Weil positions orchestration as the architectural consideration that goes beyond raw token or GPU cost calculations, because it determines whether a system can serve distributed users efficiently from a smaller number of model instances. Without orchestration capability, teams default to replication as the only available tool for managing latency, which inflates cost without improving the underlying architecture.
“You need to be thinking about how am I going to have the orchestration capability to decompose a request and go to where the right information or the right code is being managed for my use case. That is an extra layer of architectural consideration that goes beyond just how much do my tokens and GPUs cost.” — Ari Weil, VP Product Marketing, Akamai
Q: How does model freshness and continuous learning affect placement decisions?
When a model requires continuous updates or ongoing learning, replicating it across locations introduces a synchronization and data movement cost on top of the base replication cost. Weil notes that teams must account for how data moves into and out of the model, and how that data movement relates to where the GPU or CPU infrastructure is located, before deciding to use geographic copies as a latency solution. A model that changes infrequently is a reasonable candidate for multi-location deployment; a model that is continuously retrained introduces compounding cost and complexity with each additional copy.
“If I need to keep my model up to date, continuously learning, I need to think about the size of that model and how I’m moving data into and out of it, how that is going to relate to where I place the workload and where my GPU or CPU infrastructure is going to be.” — Ari Weil, VP Product Marketing, Akamai
Q: How does culture and process from DevOps and DevSecOps apply to controlling AI infrastructure costs?
The same principles that drove the shift from monolithic software to microservices, where smaller, purpose-built components outperform large general-purpose ones in cost and efficiency, apply directly to AI model selection and deployment. Weil engages with the premise that practitioners do not need massive frontier models for every use case, and that choosing a smaller, domain-specific model and designing the architecture deliberately around its placement is the AI equivalent of the operational discipline that DevOps introduced to software delivery. Cost control in AI is not purely a financial function; it is an architectural and cultural discipline that must be built into the design process from the start.
“Make sure that you understand what model you’re using for which purpose.” — Ari Weil, VP Product Marketing, Akamai
Resources & Documentation
- Akamai, cloud computing, security, and content delivery platform referenced throughout as the infrastructure context for distributed AI inference
***
👇 Click to Read Full Raw Transcript
Swapnil Bhartiya: Now I always argue that we should not look at AI any differently than we look at software. The closer it is to a user, the better performance you get. Of course, just because piece of code or software is massive or monolith doesn’t mean it is more powerful and more efficient. In most cases is smaller. That’s why you break it down. A smaller piece of code is better. Same way you don’t need large language model with massive billions and billions of parameters. Sometimes smaller models work better and they are most cost efficient as well. Another factor is culture as we have seen a lot of cultural shift. DevOps, DevSecOps, SR, all the label, all the Personas are there in terms of AI. How much role do you see culture process practices as we have adopted in the software world also play in AI world when it comes to taming cost?
Ari Weil: I mean to me we have to look at the different cost factors that we use to model our business. Many times people will look at what is my return on investment, what is my total cost of ownership, how quickly am I scaling and what is my cost of customer acquisition. Those are three numbers, sort of when you think about ROI, TCO and CAC that many people in SaaS environments are very used to looking at all the time. But if we think about where AI is a little bit different in this equation, it could be in the past, using in its recent past, but using a non AI centric workload that I would build in the cloud, that I could scale more readily by deploying additional compute, deploying my microservices to additional environments that would help me with architectural availability and resilience, but it could also help me with latency and scale up if I needed to put my microservices in more places to accommodate more users or more concurrent users. The challenge when it comes to AI is that if you think about what you would need to put in multiple locations, a lot of companies don’t have the experience a lot of developers haven’t yet figured out. As an example, can I afford to or does it make sense to deploy my model in multiple locations? Because there’s a fair amount of data and you mentioned some of these very very large models that people are used to licensing or used to referencing versus a small fine tuned model that I might use for another purpose. Now the first thing that I would say is make sure that you understand what model you’re using for which purpose. If I am an industry or a domain specific application developer and I need to make sure that my app is going to give me Quick responses, it’s not going to hallucinate and I’ve got all of the specific information that I want coded into the model that I need. I might be looking at a fine tuned or a domain or an industry specific model that typically will be smaller to than these frontier large language models because it doesn’t need to be trained on all of the same things for me to address the use case for my domain or for this use case that then also has to do with the size of the model. So if I have a fine tuned model it might encourage me or I might be tempted to deploy that in multiple places. But here’s where we can run into a challenge. You can fix a distance problem with more compute if it’s just pure and sheer compute power. If I need to keep my model up to date, continuously learning, if I need to think about the size of that model and how I’m moving, moving data into and out of it, how that is going to relate to where I place the workload and where my GPU or CPU infrastructure is going to be for it. Sometimes dealing with proximity by having copies of my model or my data in multiple places is just going to incur me a certain amount of additional cost because I’m going to pay twice for the same answer. Arguably, if I could come to one model from multiple locations where the user is making a request and I know that ultimately I’m going to be serving the same answer, but I’m copying that model to overcome a latency barrier. Now I’ve included cost just to overcome latency by copying my model. And again that could be okay if you think about my model isn’t really changing that much and I just need that proximity to maintain a latency budget. That’s a consideration that you would make at design time or at architecture time and then you can think about the cost that you would incur. But if I think about how workloads can become interactive and chatty, a lot of the interfaces for the use cases that modern AI inference are addressing are voice based, are sort of video to text based or speech to text based or real time decisions like fraud interception for example, or a voice agent. Those types of things need low latency, they need real time responsiveness. And so if you think about concentrating that sort of a workload into centralized regions, it’ll work great when people are close to that region. But your performance and your cost will suffer the further away you get because you will start to implement additional capabilities to overcome that latency. Some of which could be as we were saying before copying that model. So I think the centralized cloud thesis makes sense if you have centralized functions that are sparsely or sparingly used, that are not directly in the request response timing for an end user request, or if that request doesn’t require real time. But if you need something like anomaly detection or fraud detection, or to provide somebody with a real time query based on something where they’ve already been conditioned to anticipate a real time response, then you need to be thinking about how am I going to have the orchestration capability to decompose a request and go to where the right information or the right code is being managed for my use case? And that is an extra layer of architectural consideration that goes beyond just how much do my tokens and how much do my GPUs cost.
