By Token pricing is the number every team sees at procurement time. It is not the number that shows up on the bill. Cross-availability-zone traffic, egress fees between cloud regions, and the cost of copying model weights to overcome latency accumulate silently from the first production request. By the time an engineering team notices the margin compression, the architecture is already locked in.
In this interview on TFiR, Ari Weil, VP of Product Marketing at Akamai, breaks down the real cost structure of AI inference at scale, walks through the workloads where centralized GPU clusters break down, and explains how teams can apply distributed systems thinking to model the full bill before they ship.
Guest: Ari Weil, VP of Product Marketing at Akamai
Show: TFiR
Here is what every platform engineer and AI infrastructure team needs to know.
Technical Deep Dive
Q: Why is AI inference cost a harder problem than training cost?
Ari Weil, VP of Product Marketing at Akamai, draws a clear line between training and inference: training generates the headline investment, but inference is the bill that arrives every time the model is called in production. Unlike training, which is a bounded event, inference runs continuously across every user request, agent call, and automated pipeline. The architecture surrounding the model, not just the token price, is what determines the actual cost at scale.
“Inference is the bill that you get after you get all excited about training, and it arrives every single time your model takes a breath.” — Ari Weil, VP of Product Marketing, Akamai
Q: Why is budgeting AI applications by GPU cost and token price alone a mistake?
Weil argues that teams quoting token price are missing the architectural layer that actually determines the bill. Every time a request crosses a zone, region, or cloud boundary, the team pays twice: once in latency and once in transfer fees that are entirely separate from token pricing. The correct mental model is distributed systems architecture, not a line-item procurement decision.
“You need to stop budgeting your AI applications as line items to do with just the GPUs that you’re procuring. You need to budget like building a distributed system.” — Ari Weil, VP of Product Marketing, Akamai
Q: How much does data movement actually cost in an AI inference architecture?
Cross-availability-zone traffic runs at roughly one cent per gigabyte in each direction. Internet egress fees range from 9 to 12 cents per gigabyte. In an inference pipeline where embeddings are stored externally, microservices are called across zones, and results are returned across cloud boundaries, those charges compound on every request. Weil’s point is that these are not physics constraints; they are architectural choices that teams are billed for as though they were physics.
“Latency and egress, that’s not physics. That is an architectural choice that you’re billed for as though it were physics.” — Ari Weil, VP of Product Marketing, Akamai
Q: When does copying a model to multiple locations help with latency, and when does it just increase cost?
Distributing model copies can be justified when the model is stable, the latency budget is tight, and the workload is not continuously updated. However, if the model requires ongoing fine-tuning or continuous learning, copying it to multiple regions means paying to synchronize data and serve the same answer from redundant infrastructure. Weil frames this as a design-time decision: teams need to evaluate whether they are solving a genuine latency problem or simply incurring extra cost to overcome a placement problem they could have avoided architecturally.
“If I’m copying that model to overcome a latency barrier, now I’ve incurred cost just to overcome latency by copying my model.” — Ari Weil, VP of Product Marketing, Akamai
Q: Which AI inference workloads break down under a centralized cloud architecture?
Weil identifies real-time responsiveness as the threshold. Fraud detection, voice agents, autonomous vehicles, augmented and virtual reality applications, real-time logistics for first responders, and multiplayer online gaming all require responses that cannot tolerate the round-trip latency of a centralized region when the user or device is geographically distant. Centralized compute works well for batch processing, infrequent workloads, and functions that are not in the critical path of a user-facing request.
“The centralized cloud thesis makes sense if you have centralized functions that are sparsely or sparingly used, that are not directly in the request response timing for an end user request, or if that request doesn’t require real time.” — Ari Weil, VP of Product Marketing, Akamai
Q: How did Akamai design its inference cloud to hit low-latency SLAs at the edge?
Akamai built its inference cloud on the Nvidia hardware stack and the Nvidia enterprise software stack, federating inference across an edge footprint to achieve low single to double digit millisecond response times. The architecture places points of presence near where requests originate and near where security processing needs to occur. Heavier, slower centralized compute handles persistence, logging, and non-real-time functions, keeping those operations out of the critical response path.
“The way that Akamai has found that we are able to repeatedly hit those types of SLAs is by federating things out across our high performance network and making sure that there are points of presence to respond to those requests nearer to where the requests are being made.” — Ari Weil, VP of Product Marketing, Akamai
Q: What does a real-world end-to-end AI inference workload actually look like at the application level?
Weil uses a real estate application as a concrete example. A single user session spans online listing rendering, zero-click search optimization, open house scheduling, mortgage eligibility assessment via retrieval-augmented generation (RAG) against credit and background data, and multi-step transaction approval. Each of these sub-workflows has different latency requirements, different data residency considerations, and different risk profiles. The eligibility and approval steps, in particular, require fast, secure, distributed inference because the user is waiting in real time for a high-stakes answer.
“Those types of things where you’re sitting there interacting in real time, where you’re expecting a response for something that is very high value and typically has a pretty good amount of risk associated with it, that’s a great example and a great use case for distributed AI.” — Ari Weil, VP of Product Marketing, Akamai
Q: Why are zero-click searches and LLM scrapers forcing architectural rework right now?
Weil notes that LLM-powered search is changing how users discover content. When an LLM answers a query directly without sending the user to a website, the application must be architected so that its pages are cited by the LLM rather than simply ranked by a traditional search index. This is driving active application and infrastructure rework at companies that depend on organic discovery, such as real estate listing platforms. Core web vitals performance and LLM citation frequency are both architectural concerns, not just SEO concerns.
“Zero click searches and LLM scrapers, that’s just finding my listing. And already that has a significant amount of architectural rework and application rework that companies are going through.” — Ari Weil, VP of Product Marketing, Akamai
Q: How do egress fees function as a lock-in mechanism on hyperscaler platforms?
At 9 to 12 cents per gigabyte, egress pricing on major hyperscaler platforms is structured in a way that makes large-scale data migration prohibitively expensive. Weil points out that some vendors offer a one-time egress exemption if a customer is leaving their platform entirely, but only under that condition. The practical effect is that teams that have not modeled exit costs at architecture time find themselves locked in by the sheer volume cost of moving their data elsewhere.
“Nine to 12 cents a gigabyte is something that is designed to keep your data where it is on most hyperscaler platforms, because the cost of moving it is prohibitive.” — Ari Weil, VP of Product Marketing, Akamai
Q: What is a neutral architecture and why is it relevant to AI inference cost?
A neutral architecture stores data in a single location, federates or synchronizes it across regions as needed, and serves it from wherever makes the most sense for latency and compliance. Weil presents Akamai’s converged core-to-edge architecture as an example of this approach. The argument is that what appears to be a more expensive distributed architecture on paper can become cost-competitive once teams account for the data movement costs they are no longer paying between microservices, across geographic boundaries, and for redundant storage.
“A converged architecture like Akamai, where we have core to edge, can in some cases look like you might be thinking about an expensive distributed architecture on the face of it. But when you think about all of the costs that you don’t have to pay from moving data between microservices or other services, it could actually turn into a cost benefit.” — Ari Weil, VP of Product Marketing, Akamai
Q: How should teams use real user monitoring and FinOps tools to model inference cost before scaling?
Weil describes a layered approach: use real user monitoring tools to capture actual session data including request count, data transfer volume, and device and network type, then replay those sessions through the architecture to understand cost dynamics at scale. Akamai’s CloudTest product is one example of this capability. FinOps tooling then sits on top to normalize billing data across cloud providers and surface the cost per use case. The combination of observability, active and passive testing, and FinOps modeling gives teams the inputs they need to evaluate whether a given workload’s margin profile makes sense before they commit to an architecture at scale.
“I would take my FinOps tool, I would take my observability tool, I would take my active and passive testing tools, and then it basically becomes models and math.” — Ari Weil, VP of Product Marketing, Akamai
Q: What factors beyond raw compute pricing should technology leaders evaluate when scaling inference globally?
Weil recommends starting with user geography and use case definition, then layering in regional compute costs, data privacy regulations, data sovereignty requirements, egress and ingress fees by region, audit scope, and data retention rules. Storage modeling and database modeling are directly shaped by how long data can be legally retained and where decisions can be made based on collected data. The framework then extends to growth modeling: how many additional models will the application require, how will they be consumed, and does the architecture support incremental hardware procurement or require a full redesign to scale.
“Once you know where your users are and those use cases that you’re modeling, it’s fairly straightforward for you to understand what are the costs of compute in those regions and what are the nuances around data privacy laws and data egress and ingress fees.” — Ari Weil, VP of Product Marketing, Akamai
Q: What does good cost discipline look like for teams scaling AI inference globally right now?
Weil frames the answer around what the business is optimizing for. Developer platform companies prioritizing community growth may accept higher customer acquisition cost and lower per-user revenue in exchange for daily active use and mindshare. Enterprise application companies need to anchor on unit economics: cost per seat, monthly and annual recurring revenue, customer acquisition cost, and lifetime value. The inference architecture that makes sense for a growth-stage developer platform is a different architecture than what an enterprise SaaS company needs to maintain margin at scale. There is no universal answer; the right architecture follows directly from what the business is measuring.
“It depends what you’re optimizing for. If it’s mind share and growth and consistent usage, then you really need to prioritize those numbers. If it’s margin based, then you need to think about unit economics and scale, and it’s a different architecture altogether.” — Ari Weil, VP of Product Marketing, Akamai
Q: How does the culture and process discipline from DevOps and cloud-native development apply to controlling AI costs?
Weil argues that AI inference does not fundamentally change how distributed applications are built and measured; it adds a new capability with a specific size, cost, and set of associated workloads. Teams already familiar with ROI, total cost of ownership, and customer acquisition cost frameworks have the tools to apply those metrics to AI workloads. The gap is in recognizing that a frontier LLM or a fine-tuned model is a component in a distributed system, not a category apart from it, and that the same architectural discipline used in cloud-native microservices applies directly to inference workload design.
“AI might change the game in the capability that we’re enabling. It doesn’t necessarily fundamentally change the way that I’m building and distributing an application. It’s just adding a new capability to it.” — Ari Weil, VP of Product Marketing, Akamai
Resources & Documentation
- Akamai Inference Cloud, Akamai’s edge-distributed AI inference platform built on the Nvidia hardware and enterprise software stack, designed for low single to double digit millisecond response times
- Akamai CloudTest, real user monitoring and load testing product for capturing and replaying user sessions to model cost and performance at scale
- Nvidia AI Inference, hardware and enterprise software stack referenced as the foundation of the Akamai inference cloud deployment
***
👇 Click to Read Full Raw Transcript
Swapnil Bhartiya: Today, most companies, or most of us, think that the hard part of AI is building the model. It is not. The hard part is running it in production without cost spiraling out of control. You have to look at egress, surprises, latency wall, GPU capacity that you cannot predict or plan for. And here is what makes this even worse. Most teams cannot even track whether their AI workloads are scaling efficiently or are just getting more expensive. And today we have with us Ari Weil, VP of Product Marketing at Akamai, to break down the real economics of AI inference and why where you run it changes everything.
Ari Weil: Absolutely. So, first of all, it’s great to be back on the program with you again, and I’m glad to have a chance to talk about the cost side of the equation when we think about AI applications and workloads. And I think to start off with, I would suggest that training is really the headline that everybody loves. It’s driving these massive data center deals, big investments, and some of the circular investment that we’ve seen coming out of anything from chip manufacturers and card manufacturers to the major hyperscale clouds that are also building out their own infrastructure and procuring it from others. And even if we talk about the landscape of companies that are looking to do their first IPO, a lot of the focus has been on how large of a data center they’re building out to build the next multi billion, multi trillion weight model. I think that training and that headline sort of factor is real. But if we think about it, inference is the bill that you get after you get all excited about training and it arrives every single time your model takes a breath, every time that you’re not training, either pre training or post training or fine tuning that model. Inference is what happens when somebody needs your intelligence to be deployed. And so I think the highlight from our perspective, where we would start a conversation, is to say you need to stop budgeting your AI applications as line items to do with just the GPUs that you’re procuring or just the applications that you’re building. You need to budget, like building a distributed system, which is something that many large organizations have familiarity with, or at least there are representative architectures for in the marketplace, because you’re going to hear people quote token price, how much does it cost per token for me to generate a workload? But the architecture is what ultimately, ultimately is going to decide your bill. And that’s the part that people need to get more conversant on and take some of their cues from building cloud native systems and to some extent edge native systems for the last several years.
Swapnil Bhartiya: When it comes to cost, most of the times when we look at and you will see, oh, output token, $5 sorry, input token, $5 output token. Depending on what you’re using. If you’re using Opus or Gemini Pro, you’re looking at $130 per million token output tokens. But the thing is not as simple as that. There’s a context window. The more you use it, those context, those pile up. So you even if you’re sending five messages, it will also look at the last 20 messages. So that cost is not the message you send, it is your message plus five other message. Sometimes we use. Sometimes running it locally on a local machine is the cheapest and the most perfect best way. No latency. But not everybody can run it on their local machine. Can you talk about what role does physical location of AI inference matter or play a role when it comes to performance latency and most importantly, cost. Because cost is going to become a very big factor. Because as there are reports coming out, it’s actually cheaper to hire an employee than to use AI.
Ari Weil: That is true. I mean, the first thing that I would start with, and you’re absolutely right about the way that some people are modeling applications, distance does come at a cost. And a lot of times you’re going to price that cost at zero. Because when you’re modeling your application, you’re thinking about things like what is the workflow that I’m enabling? Or what is the output of a given function or a given prompt supposed to be? And really focusing on that. But every time a request crosses a zone or a region or a cloud boundary, you’re going to pay twice. Effectively, the first time you pay, you’re going to pay for latency. How much is my time to first token? What is the overall overall time that I’m going to expend consuming all of the tokens from my request and response? The other one is the transfer fees that basically are not part of your token price. It’s what you’re charging anytime that you need to move data either between microservices, overall services or other cloud boundaries. And if you think about that latter piece, cross availability zone traffic can run a penny per gigabyte in each direction. So if I’m going to be sending data from a model to a set of GPUs that are deployed in a different part of my architecture, and then as part of that workload, I’m going to be sending either some of my embeddings back to store, or if I have to go and call other services along the way, I am going to pay again, penny per gigabyte each time that I send that data across. Egress fees then if I send them across the Internet are 9 to 12 cents each. So there’s a sort of tension between how do I build my application to take full advantage of the cloud capabilities that I know how to build on, or the different providers are making available. Does it make sense for me to keep my workload on a given cloud or where I have to cross cloud boundaries, am I ready to incur that 9 to 12 cents per gigabyte? Understanding how my app needs to scale so things like latency and egress, that’s not physics, that’s not the speed of light. That is an architectural choice that you’re billed for as though it were physics. The physics is what you have to think about. And again, where people are building AI applications and afraid that they don’t have a frame of reference for what they’re building, they can look at the last 10 to 15 years of building distributed cloud apps and understand what does data movement cost me and then really start to think to your point, when I move from a prototype to production, how many zones do I have to move from and to where does my application need to be? And one of the levers that we treat late in that cycle, late in that consideration phase, is cost. And I would argue that you have to really bring that into a lot of your user and scalability testings before you launch your app.
Swapnil Bhartiya: Now, I always argue that we should not look at AI any differently than we look at software. The closer it is to a user, the better performance you get. Of course, just because piece of code or software is massive or monolith, doesn’t mean it is more powerful and more efficient in most cases is smaller. That’s why we break it down. A smaller piece of code is better. Same way you don’t need large language model with massive billions and billions of parameters. Sometimes smaller models work better and they are most cost efficient as well. Another factor is culture, as we have seen a lot of cultural shift. DevOps, DevSecOps, sorry, all the label, all the Personas are there in terms of AI. How much role do you see culture, process practices as we have adopted in the software world also play in AI world when it comes to taming cost?
Ari Weil: I mean to me we have to look at the different cost factors that we use to model our business. Many times people will look at what is my return on investment, what is my total cost of ownership, how quickly am I scaling and what is my cost of customer acquisition? Those are three numbers, sort of when you think about ROI, TCO and CAC that many people in SaaS environments are very used to looking at all the time. But if we think about where AI is a little bit different in this equation, it could be in the past, using in its recent past, but using a non AI centric workload that I would build in the cloud that I could scale more readily by deploying additional compute and deploying my microservices to additional environments. That would help me with architectural availability and resilience, but it could also help me with latency and scale up if I needed to put my microservices in more places to accommodate more users or more concurrent users. The challenge when it comes to AI is that if you think about what you would need to put in multiple locations, a lot of companies don’t have the experience a lot of developers haven’t yet figured out. As an example, can I afford to or does it make sense to deploy my model in multiple locations? Because there’s a fair amount of data and you mentioned some of these very very large models that people are used to licensing or used to referencing versus a small fine tuned model that I might use for another purpose. Now the first thing that I would say is make sure that you understand what model you’re using for which purpose. If I am an industry or a domain specific application developer and I need to make sure that my app is going to give me quick responses, it’s not going to hallucinate and I’ve got all of the specific information that I want coded into the model that I need. I might be looking at a fine tuned or a domain or an industry specific model that typically will be smaller than these frontier large language models. Because it doesn’t need to be trained on all of the same things for me to address the use case for my domain or for this use case that then also has to do with the size of the model. So if I have a fine tuned model it might encourage me or I might be tempted to deploy that in multiple places. But here’s where we can run into a challenge. You can fix a distance problem with more compute if it’s just pure and sheer compute power. If I need to keep my model up to date, continuously learning, if I need to think about the size of that model and how I’m moving data into and out of it, how that is going to relate to where I place the workload and where my GPU or CPU infrastructure is going to be for it. Sometimes dealing with proximity by having copies of my model or my data in multiple places is just going to incur me a certain amount of additional cost because I’m going to pay twice for the same answer. Arguably, if I could come to one model from multiple locations where the user is making a request, and I know that ultimately I’m going to be serving the same answer. But I’m copying that model to overcome a latency barrier. Now I’ve incurred cost just to overcome latency by copying my model. And again, that could be okay. If you think about my model isn’t really changing that much and I just need that proximity to maintain a latency budget. That’s a consideration that you would make at design time or at architecture time, and then you can think about the cost that you would incur. But if I think about how workloads can become interactive and chatty, a lot of the interfaces for the use cases that modern AI inference are addressing are voice based, are sort of video to text based, or speech to text based or real time decisions like fraud interception for example, or a voice agent. Those types of things need low latency, they need real time responsiveness. And so if you think about concentrating that sort of a workload into centralized regions, it’ll work great when people are close to that region. But your performance and your cost will suffer the further away you get because you will start to implement additional capabilities to overcome that latency, some of which could be as we were saying before copying that model. So I think the centralized cloud thesis makes sense if you have centralized functions that are sparsely or sparingly used, that are not directly in the request response timing for an end user request, or if that request doesn’t require real time. But if you need something like anomaly detection or fraud detection, or to provide somebody with a real time query based on something where they’ve already been conditioned to anticipate a real time response, then you need to be thinking about how am I going to have the orchestration capability to decompose a request and go to where the right information or the right code is being managed for my use case. And that is an extra layer of architectural consideration that goes beyond just how much do my tokens and how much do my GPUs cost.
Swapnil Bhartiya: One thing more is that a lot of organizations, they are kind of used to thinking about AI infrastructure in terms of centralized big cloud and huge massive GPU clusters where does that model start to break down when AI gets embedded into real time applications?
Ari Weil: So if I think about things like real time personalization, if I’m in a retail or an e-commerce sort of a workflow, my anticipation is that as I click, there’s going to be a response from the application, whether that’s a web application or a native mobile application, or even if I’m doing it through some sort of an agentic platform. So real time is real time. If I think about things like fraud detection I’ve mentioned a few times now, those things need to happen incredibly quickly because of the pace and the frequency of financial transactions when themselves. If I think about a voice agent or something that is doing real time logistics, for example, to send first responders to where they need to go and manage traffic events while they’re going to their first response or responder locations, these are all things where something is waiting on that answer. And that answer can’t be a continent or a country away, it needs to be nearby. And so if I think about the common use case of you fan out to try to go out and find the resources that you need, get all of the tools, get the data, get the context that’s required, and then you fan in. The heavy part of that equation is that last mile of reaching the device that made the request in the first place. And those are sort of the things that the centralized model handles the worst. Because fanning out, fanning in and doing so quickly is something that works well in a centralized location for batch processing or infrequent processing. But the reason that we built Akamai inference cloud using the Nvidia hardware stack, using the Nvidia enterprise software stack, was that we needed to run inference across an edge footprint that supported low single to double digit millisecond response times. And the way that Akamai has found that we are able to repeatedly hit those types of SLAs is by federating things out across our high performance network and making sure that there are points of presence to respond to those requests nearer to where the requests are being made and also nearer to where we have to apply some of the security considerations, leaving that heavier weight, slower centralized compute to be things like persistence and logging and the sorts of things that are not going to get in the way of that real time answer that either your robotics, your physical AI or your humans are looking for responses to.
Swapnil Bhartiya: Are there specific workloads or use cases that make the strongest case for running inference closer to users, devices or data sources, or you feel that every use case, every workload is ideal if it is running closer?
Ari Weil: Oh no, definitely not. I mean if I were going to be training a model, pre training or post training of my model to do anything from adding weights to the model or embeddings that I might need later, if I’m training, paring down a model that I might use in a future workload, and I want to make it a smaller overall model for storage reasons and what I can hold resident in memory, we don’t need those things to be federated. Those don’t have to be real time and responsive in this way. If I’ve already got a model and the use case is, hey, I have an LLM backed mortgage application or mortgage approval application, then I can think about when the user logs in, they’re going to provide me a certain amount of data, I might want to access my data about that user and maybe I’ll perform some RAG activity to do a credit check or to do a background check on that person. But all of it needs to come together in under, you know, five seconds, typically certainly under 30 seconds and provide me with an answer that says you are or not approved for this loan. You are approved for this much and here are the next steps to actually capitalize. And if I think about that in a workload, like an end to end workload sort of perspective, imagine that I am a real estate company and I might have anything from my online listings. I need to make sure that all of my listings render completely correctly, consistently. But certainly that quickly piece needs to be top of mind for me. I need to ensure that in LLMs any zero click searches are actually having my page be cited more frequently than my competitors page so that people find me in the first place. Just finding that real estate listing already has a significant amount of right now architectural rework and application rework that companies are going through because of this phenomenon of zero click searches and LLM scrapers. Now that’s just again finding my listing. Next thing is maybe I want to understand can I set up an open house time? Can I schedule a walkthrough with the realtor? Maybe I want to find out what it might take for me to actually get approved for a loan on this property. Those are different work streams that I’m building into my application and each of those as well needs to be real time. You would argue that scheduling something on the realtor’s calendar is a pretty lightweight use case and you would be right. There’s not much of an appeal for AI in that sort of a use case. But confirming somebody’s eligibility and what level or what size of loan they could get is very much something that AI can help with. And it’s very common in the financial services industry, specifically around insurance and warranties and mortgages that we need to run quickly, securely and get an answer back to that user before they lose interest and walk away, or before they worry that maybe their information was compromised. So now I’ve got a listing that I’ve received, I’ve applied for a loan, maybe I’ve set up time for an open house. The next thing that I might consider is the actual set of transactions where I want to procure the home or I want to finalize that mortgage, I want to purchase the house, I want to go through other signatories. Those types of things typically as well will require some form of multi step approval process that is currently being automated by the people that are managing it. And so those types of things where you’re sitting there interacting in real time, where you’re expecting a response for something that is very high value and typically has a pretty good amount of risk associated with it, that’s a great example and a great use case for distributed AI because you are running inference and you want those answers to be fast, reliable and secure. If I think about some other ones that are related, you know, anytime that I might do anomaly detection, that could be in an autonomous vehicle, it could be in a robot or other physical AI device, it could be for augmented reality. If I’m doing work on a conveyor belt or an assembly line, I might be in the medical field, I may be in the public sector or even in a military sort of a facility. But there’s real time inference happening through augmented reality and virtual reality goggles today that lend themselves to this. And then we can think about, you know, even simpler things like real time multiplayer online gaming. You’re creating worlds, you’re having avatars and other characters in the game that are reacting and responding to people in real time. These types of things you keep hearing real time or where you have a physical engagement with the application, those are typically the things that would warrant distribution to maintain that low latency and that real time performance. But I think we’re just starting to understand this as an industry and even things like how we’ll create online sports broadcasting, news broadcasting, gambling, e-gaming, things that really do reward latency and instantaneous responses are where we are seeing companies invest in this area and start to experiment with new ways of shuttling data across systems and starting to think really deeply about where does the hardware go, where does the data go, including the models, and then where is that application experience going to be interacted with. And that’s what’s driving the considerations.
Swapnil Bhartiya: You touched upon egress cost earlier. When teams move AI into production, egress costs often come as a surprise to them. How much of that hidden cost of centralized inference is actually just data movement getting results back to users and applications? How big of a factor is that and how can teams plan for that as well?
Ari Weil: So I think, you know, moving into the next 12 to 36 months, the companies that are going to grow the fastest and the most successfully will have modeled the whole bill and not just the line item that the hyperscalers are touting or that all of the different AI platforms are touting. They need to really understand all of the use cases that will incur cost for them as they scale. And that can be anything from what is my user acquisition cost, what is my concurrency cost with multiple use cases and users, how many use cases have I enabled on this platform? Where is some of that data that I own completely, my intellectual property, my systems of record, and where am I learning actively and developing from the world or the industry at large and my customer base? Because the first thing that I can do is I can understand what would that cost be if I said I don’t want to use multiple cloud providers or even my principal cloud provider, and I want to manage the data in my own environment? My own environment could be my own data center, it could be leased space, it could mean a lot of different things. But that’s the first question of what if you just owned all of your data in your application code that will have you start to understand availability zones and data boundaries, and where distribution can become more expensive if you use your own versus using a hyperscaler’s infrastructure. The second thing is once you’ve decided to partner with somebody, what would that cost to leave? Because now, if you look at all of the last 12 to 18 months of news around the cloud space, there have been all sorts of news stories about how much does it cost to leave a cloud provider? What does it cost for companies to take over their own data to export or download it themselves? And so the reality is, 9 to 12 cents a gigabyte is something that is designed to keep your data where it is on most hyperscaler platforms, because the cost of moving it is prohibitive. Some vendors have said if you’re leaving their cloud altogether, then they will give you a one time exemption where you can download your data. But you should be making sure that you indeed intend to exit that cloud environment. It’s also given rise to something that people are considering to be more of a neutral architecture. So a neutral architecture is one that will allow you to store data in one place that might then federate or synchronize it in a bunch of different areas, allow you to retrieve it somewhere else and then serve it where it makes the most sense. So a converged architecture like Akamai, where we have core to edge, can in some cases look like you might be thinking about an expensive distributed architecture on the face of it. But when you think about all of the costs that you don’t have to pay from moving data between microservices or other services, moving it across geographic boundaries, and what it would take to manage all of your data and application code and then serve it as needed on demand with that low latency could actually turn into a cost benefit for you if you’ve modeled out all of your use cases across the platform. And I think that’s the ultimate question here is where does inference run for your application? How often and where do you need to do things like pre and post training and fine tuning of your model? And then where is the rest of the activity in your application actually occurring? Because if you know those things and you know how you’re modeling user acquisition and user growth and long-time value, then you can put all of those factors together and think about what sort of an architecture am I really focused on and which parts of my application can be centralized versus where I’ll benefit from distribution.
Swapnil Bhartiya: Now let’s talk about tracking and measuring. If most organizations can’t even track unit economics for inference, how do they even know whether their AI is scaling efficiently or just getting more expensive?
Ari Weil: Yeah, so I think in some cases I’ll use an example from Akamai. We have a product called CloudTest. That product is designed to look at the real user monitoring of all of the users of your application, your SaaS application in the case of a proxied app on Akamai, and to show you how many people are using it, where are they coming from, what type of devices and network connections are they, how many requests and responses are part of an average user session. And then you can start to replay those things through your cloud and your edge network so that you understand exactly the dynamics. Whether you use a Wireshark to look at the network connections and the network dynamics, or you use another active or passive testing tool to look at those real user activities and replay them and start to decompress them into repeatable steps that you might use in a testing harness. But there are multiple ways to get whether you’ve architected the whole user interaction or you’re just recording the reality of a user interaction coming to you. All of those individual steps, the systems that you hit, the amount of data that you typically transfer and then it becomes a modeling exercise. You can do it virtually or you can do it maybe conceptually in spreadsheets and databases and actually use math just to extrapolate out this is what my growth curves will look like. You can use real user examples like that CloudTest product that I mentioned so that we can look at the volume of users coming to you. We can look at their recorded sessions and actually show you what that user interaction looks like. And there’s a myriad of other observability tools that can help you do something similar. There are also FinOps tools that you can bring to bear to help you make sense of the cloud bills that you’re seeing, including if you have, or maybe especially if you have a multi-cloud environment where the way that things are being metered and billed are different. So I would take my FinOps tool, I would take my observability tool, I would take my active and passive testing tools and if you’re an Akamai customer, we have a number of these for you. You can also use services if you want to to help you with sort of this modeling and observing your actual user workloads. But then it basically becomes models and math. It’s how much does it cost for me to enable this use case? Is that use case as optimized as I might want it to be, how much does that cost me? And then by channel you can start to understand with your revenue operations or product or business operations teams, am I incurring a cost that’s commensurate with what I’m making? And does the margin profile for this channel or this workload or this application make sense? And I think every business has the tools to do this. It’s just when we look at AI, a lot of times we’re thinking that this special thing that is a frontier LLM that maybe we’ve licensed, or maybe it’s a fine tuned or a post trained model that we’ve created ourselves somehow completely changes the game. It might change the game in the capability that we’re enabling. It doesn’t necessarily fundamentally change the way that I’m building and distributing an application. It’s just adding a new capability to it. But that capability has a size and it has a cost, and it has a set of activities that workloads spawn from it or to it. And you can measure and meter all of those things to come up with the right architecture and the right scalability model for your business.
Swapnil Bhartiya: For those businesses and technology leaders who are trying to manage AI costs, what should they be evaluating beyond raw compute pricing?
Ari Weil: Yeah, I think the raw compute is certainly important. Understanding if I expect my model to grow and how much data that it might be sending out from various queries is also important. But I would argue you could start simpler than that and just say, what are the core use cases for my app? Where do I anticipate my users to be? Most companies, most product teams, are used to looking at user stories that define an ideal customer profile. You’ve got demographic and firmographic information about who those users are that you’re looking to attain. If you’ve done a tech preview, an alpha or a beta release, you’ve probably got an idea of how you might need to fine tune some of those assumptions around ICP and target account lists and things of that nature. But once you know where your users are and those use cases that you’re modeling, I do think that it’s fairly straightforward for you to understand what are the costs of compute in those regions that I’m trying to serve? What are the nuances that I need to understand about data privacy laws, data egress and ingress fees that I might be accumulating when I want to do business in that region? And then you can start to ask yourself, how often do I have to think about things like being audited and what might be the scope for that audit, what sort of privacy regimes or sovereignty regimes I may need to sign up for, which might specify how my architecture is set up, where I need to persist data, where I’m allowed to actually make decisions based on the data that I’ve been collecting, how long I can save that data, because all of that will have an impact for storage modeling and database modeling. And then you need to think about how am I thinking of growing the capabilities of this application over time, and what does that need to grow? Do I just need additional hardware so that I can process more off of the existing LLMs or SLMs that I’m managing? Do I need to continuously deploy more models for these capabilities, and are they going to be consumed by all of the same users the same way? And so to me it becomes really a microservices architecture scaling exercise. And between the FinOps teams that have been helping cloud practitioners understand what sort of costs they’re incurring, giving way to some new versions of FinOps that now are taking into account AI considerations, billed by partner, and also how you would architect the AI request response calls. As people are getting more examples and more comfortable with AI applications, we’ve got a lot of the raw materials for this. But the decisions ultimately will be what do I have, where are my users, and what does it cost for me to make those two things connect?
Swapnil Bhartiya: If I ask you what does good cost discipline look like for teams that are scaling inference globally right now?
Ari Weil: I mean, that’s a great question. I think if I were to think about my existing architectures and how our teams are used to looking at them, typically the question is going to be if I said that I was prioritizing one thing, it would be growth, right? I want my user count to grow, I want my active users to grow, and I want my revenue to grow. The number one factor will ultimately depend on what kind of a business you are and what you prioritize. Some businesses need to grow a large community to have any hope of scaling and becoming successful. I think about some developer platforms out there. Unless you have the support of a large group of developers, there’s simply no way for you to be successful because ultimately developer sentiment and developer mindshare drive you. And so in those cases, maybe my cost for customer acquisition can be a bit higher. And off of some of the centralized indexes that have become standard, maybe the ultimate revenue that I get generated by user will be a little bit smaller. But if I look at customer acquisition and daily active use, those are the optimal metrics that I’m trying to prioritize. If I look at enterprise applications, it probably comes back down to margin. And there I’m thinking about how do I procure more users, how much are they paying me by user, perhaps by seat or by month or year. So I’m looking at MRR, ARR, or I’m looking at cost per user, cost per seat, and then I’m thinking about how much does it cost for me to serve them an experience, depending on what that experience is. And does that math balance, like what is my CAC and what is my LTV? And from that can I get the ROI from all of the sales and marketing that I need to invest to get and keep people on that platform. So I think the answer varies. I always think about my favorite architect in Australia when I was a young product manager and he used to always ask me how long is a piece of string when I’d ask a non-deterministic question. And so I think that is sort of the answer to your question, Swapnil. It depends what you’re optimizing for. If it’s mind share and growth and consistent usage, then you really need to prioritize those numbers and the way that you’re updating your application to drive that engagement. If it’s margin based because you’re an enterprise app or otherwise somebody that needs to scale up to maintain a consistent cost of service with a consistent benefit from the service that you’re providing, then I think you need to think about unit economics and scale and it’s a different architecture altogether. But that’s such a use case and company specific question that it really does depend on what you’re building for.
Swapnil Bhartiya: Ari, this is exactly the conversation enterprise teams need to be having right now. Thanks for breaking it down and sharing with our audience and folks, please head over to Akamai and check out their latest blog post where they are going to talk about cost and other challenges related to AI. Thanks for watching and we’ll see you in the next episode. Thank you, thank you.
