By Enterprise AI inference costs are scaling faster than budgets, and the standard fix, sending more context to a hosted third-party model, creates a second problem: private, regulated data leaving the perimeter. At the same time, agentic workloads are reaching infrastructure they were never authorized to touch, and most teams have no runtime boundary in place to stop them.
In this interview on TFiR, Jon Alexander, SVP of Products at Akamai, covers how Akamai’s new managed inference platform and WebAssembly-based Agent Sandbox address all three failure modes: cost, data control, and agent containment.
Guest: Jon Alexander, SVP Products at Akamai
Show: TFiR
Here is what every platform engineer and AI infrastructure team needs to know.
Technical Deep Dive
Q: Why are enterprise AI inference costs running out of control and what is driving teams to look beyond hosted model providers?
Jon Alexander, SVP of Products at Akamai, identifies three compounding pressures: inference cost scaling with usage, loss of data control when private context is sent to third-party services, and limited ability to innovate when a vendor controls the full runtime stack. Most enterprises currently rely on AI labs or managed infrastructure providers for the simplest path to frontier intelligence, but that convenience carries a per-token cost that compounds fast as adoption grows. Moving from closed-source to open-source models provides near-frontier intelligence at materially lower per-token cost while keeping data under the team’s own governance policies.
“These costs are really running away as they’re driving more adoption and more usage, more use cases within their enterprises.”
Jon Alexander, SVP Products, Akamai
Q: What actually changes on day one when a team migrates an existing application to Akamai’s inference platform?
Because Akamai’s inference platform exposes an OpenAI-compatible API, the migration is a single line change: update the endpoint to point at Akamai and add the corresponding token key. No application rewrites are required. Alexander notes that adding an AI gateway such as LiteLLM or Agent Router on top of that provides model-level and per-user routing logic, including budget-based routing that can automatically downgrade a user to a lower-cost model once a spending threshold is hit.
“It’s one line of code that has to change to switch to start using the Akamai inference platform.”
Jon Alexander, SVP Products, Akamai
Q: Why does separating an agent’s runtime from the code it generates matter, and what does WebAssembly provide that other approaches do not?
Agents that generate and execute code need a hard boundary that prevents one agent from accessing another customer’s data and prevents unexpected lateral movement to external services, MCPs, or APIs. WebAssembly provides that boundary with strong isolation, fast cold-start times measured in milliseconds, and a lightweight resource footprint that keeps cost low for short-lived ephemeral tasks. Alexander traces this capability to Akamai’s acquisition of Fermyon, whose WebAssembly functions platform was already running at the edge for personalization, bot detection, and content transformation workloads before being extended to agentic runtimes.
“The sandbox provides a boundary where we can enforce those controls. We can be very explicit around whether that agent has access to the Internet, to other services, to MCPs, to APIs.”
Jon Alexander, SVP Products, Akamai
Q: When should a team choose dedicated GPUs over serverless inference, and what performance guarantees does dedicated capacity actually provide?
Serverless inference pools capacity across customers and optimizes for utilization, meaning workloads queue during contention and performance is not guaranteed. Dedicated GPU pools give a customer reserved capacity that is never shared, eliminating queuing and guaranteeing time-to-first-token and inter-token latency SLAs. Alexander’s guidance: teams that understand their workload profile, have a defined peak tokens-per-second requirement, or need predictable SLAs for production applications should choose dedicated; teams in early experimentation or with unpredictable traffic are better served by serverless. Akamai’s fleet now includes Blackwell Ultra clusters with 288 GB of HBM3 memory per GPU to support models from small language models up to trillion-parameter frontier models.
“Dedicated GPUs are a good way of them achieving that time to first token, the inter token latency that their businesses demand, and the ability to scale up to the peak capacity levels that they require.”
Jon Alexander, SVP Products, Akamai
Q: How should developers choose between WebAssembly, micro VMs, gVisor, and container-backed agent runtimes for a given workload?
Alexander frames the decision around two axes: duration and compute access. WebAssembly sandboxes are optimal for ephemeral workloads that run for milliseconds to a few seconds: they cold-start nearly instantly, consume minimal resources, and tear down cleanly, making them cost-effective for high-volume agentic tasks. Micro VM and container-backed runtimes such as gVisor or Kata Containers are heavier, slower to start, and more expensive, but they expose a full file system and computing environment, support headless browser use, and can run for hours, days, or longer. Akamai plans to launch a container-backed agent runtime in the first half of 2027 with stateful backing for workloads that require that longer duration and richer environment.
“Depending upon the modality, the type of agent that you’re looking at, you’re really picking one of these types of presentation.”
Jon Alexander, SVP Products, Akamai
Q: What capabilities should teams building agent applications expect from Akamai’s agent security roadmap?
The next priority after the sandbox boundary is observability: ingesting telemetry from millions of short-lived sandboxes at a volume that is impossible to inspect manually. Alexander describes the roadmap as using AI itself to process that signal stream, identify unexpected behavior, determine where to apply policy changes, and decide when to destroy a sandbox or escalate to a human. The model is layered defense: start with restrictive sandbox access controls, then feed sandbox telemetry into an intelligence engine that learns normal behavior and surfaces anomalies. He is explicit that relying on the sandbox boundary alone is not sufficient.
“Inspecting these manually by human is impossible. This is where we have to use AI and intelligence to inspect all of this data being generated and to derive insights from that.”
Jon Alexander, SVP Products, Akamai
Q: How does fragmentation across neoclouds, gateways, and agent runtimes create gaps in AI governance, and what is Akamai’s integration approach?
Current AI infrastructure is assembled from components built by separate vendors: neocloud GPU providers, CPU-based agent runtime providers, agentic harness software, observability tools, and gateways. Because these components were not designed to interoperate, teams lack seamless observability across the full stack and cannot apply consistent threat models across all runtimes and infrastructure. Alexander’s stated goal is a modular integration layer where teams can bring their own gateway, run agents on Akamai while GPUs run elsewhere, and route a multi-cloud hybrid environment through one common observability and threat-model platform. The explicit design constraint is avoiding a forced full-stack commitment to Akamai.
“Only when we kind of have that ability to connect the different pieces together can we really get the true value.”
Jon Alexander, SVP Products, Akamai
Resources & Documentation
- Akamai, managed AI inference platform and WebAssembly agent sandbox for enterprise AI workloads
- Spin (Fermyon), open-source WebAssembly framework underpinning Akamai’s agent sandbox and functions platform
- LiteLLM, open-source AI gateway for multi-provider LLM routing, budget controls, and observability
***
👇 Click to Read Full Raw Transcript
Swapnil Bhartiya: A team shipping and AI feature needs more than just a model. It needs serving infrastructure, access to its own data, routing, governance, and a hard boundary around what agent written code can reach. Most teams assemble these pieces from separate providers and secure the siems themselves. Akama is making it easier. It is adding the two missing layers, a managed inference services infrastructure and a webassembly sandbox for agents that write and run their own code to unpack both. Today we have with us once again John Alexander, SVP of Cloud Technology at Akamai. John, it’s great to have you on the show.
Jon Alexander: Yeah, it’s Rob. Great to see you again. Thanks for having me back.
Swapnil Bhartiya: If you look at Akamai and based organizations, Akamai’s AI strategy was built on the thesis that inference now training would take the majority of enterprise focus and investment. And that turned out to be true. Your customers proved that thesis and pulled you. Can you talk about what were they still assembling themselves and why you’re adding managed inference and a sandbox now?
Jon Alexander: Yeah, exactly. That’s exactly right. So it’s 18 months ago from now that we actually kind of made the decision to start investing into building out Akamai’s AI stack. And as you said, we saw this transition happening where the industry was moving focus from training these very large language models into deploying inference. And inference is where we’re deploying that intelligence. We’re using it in production to drive business outcomes. And this is kind of what we’re seeing with customers now. Most of our customers are heavy consumers of AI, but constrained it in those use cases. So when we talk to our enterprise customers, the people who are buying our CDN and security services today, they’re using AI probably as an encoding assistant. They might be using it in some of their product experiences for things like natural language search. Maybe they’re using it in some of their recommendations. But it’s limited kind of deployment. And part of that is because it’s complex to build and deploy this all yourselves. And so they’re typically purchasing from one of the AI labs from one of these managed infrastructure players who’s giving them the full stack. And that’s the easiest way to get started and get access to frontier intelligence. So that’s where we see most of the industry is at today. But there’s a couple of challenges with that. So three main challenges that we see our customers facing which led to us investing in these products ourselves. First, one, probably no surprises around cost. This is not an insignificant investment that many of our customers are facing and these costs are really running away as they’re driving more adoption and more usage, more use cases within their enterprises. And so customers are starting to look at ways of managing that cost and to control how much they’re spending on AI. So moving from closed sourced models to open source is one way of getting access to near frontier level intelligence at a much lower per token cost or per unit cost. So that’s the first piece. The second piece is around data. A lot of the inference workloads that we’re seeing today only drive useful results if you pass the model. Access to your private secret enterprise information, your core business intelligence, you need to pass all that context to the model so it can make the right decisions. Make a decision whether to recommend product A versus Product B based on all the information you have around the user, around the insights you’ve gathered during your business. That data is incredibly important to your business. It’s also often data that you want to keep close control over. This is personally identifiable information that is heavily regulated and that you need to control. And sending that off to a third party service in the cloud means that you’re losing some of those restrictions. So the second piece that we’re seeing is customers want to take more control over their data, want to be able to retain the localization and where that data resides. The third piece then is really around innovation, the pace of innovation, the ability to control the full stack and to innovate on the product features and capabilities at the pace they want to. And again, on a hosted third party service where you don’t have ability to fine tune all of the runtime settings, you’re limited in how much innovation you can really drive. So these were the three things we were hearing from our customers and that’s what we set out to solve with these two new launches that we’re talking about.
Swapnil Bhartiya: As you mentioned, akamai inference serves OpenVait and custom models through OpenAI compatible APIs. So existing applications move without rewrites. For a team that’s already working on it, that already has an application in production, what actually changes on day one?
Jon Alexander: Yeah, this is one of the interesting things. So we mentioned the OpenAI compatible API interface. So one of the nice things is it’s one line of code that has to change to switch to start using the Akamai inference platform. And so rather than pointing at another service, change that one line to point at the Akamai API endpoint. Add the token key from Akamai and you’re immediately routing your token request to Akamai versus another platform makes it very, very easy to get started. You can add more intelligence. So this is where AI gateways are important. Akamai offers an AI gateway, but there’s many other options available in the market. There’s great open source solutions like Agent Router or Light LLM where you can add more sophistication around how you route tokens and so you can do model level routing. So you can say, hey, if a user is using model X, I’m going to send that to Akamai. Model Y, I’m going to send that to another provider. You can even add per user logic. So maybe I’ve exceeded my budget for the month. So in the, in the gateway you can set a policy where you can route me to another model on a different provider because I’ve exceeded my budget. So I don’t get access to that frontier level model anymore. I’m going to get access to a medium sized model where it’s lower per token cost. So there’s a lot of intelligence that you can add. That AI gateway is one of the great ways to manage which inference platform you’re sending requests to and to add this intelligence. So this is one of the things that we encourage our customers to have. Control gives you observability and visibility. You can see what’s going on, but then you can make these routing decisions. You can add intelligence to decide whether to send to provider A, provider B, and to apply those routing decisions a very granular, fine grained levels.
Swapnil Bhartiya: As you explained, Agent Sandbox puts a webassembly boundary between an agent and the code it generates separated event from the agent’s own runtime. Why does that separation matter? And why ship it as an open source component of SPIN rather than keep it proprietary?
Jon Alexander: Yeah, well, I mean this goes back to one of the things that we talked about swap, maybe a year ago. It was back in December, Akamai acquired a company called Fermyon who provided webassembly functions. And this became kind of a core part of what we call the Akamai functions platform. So webassembly provides a very nice sandboxing isolation paradigm on top of a compute infrastructure that originally we were using to deploy functions on the edge of our platform for things like personalization, recommendations, doing transformations for content as it was flowing through our CDN network. It allows us to extend our security controls for things like bot detection and additional logic around authentication and authorization. But that core technology is the great foundation for something like an agentic runtime. And so when we think about an agent. A lot of these are ephemeral, short lived. They need to spin up, do some work and then spin down. You want these to be isolated from each other. You don’t want one agent to be able to access the data of another customer. So having isolation between the agents is important. So this is the first thing that the sandbox provides. It isolates what that agent can get access to. It provides that boundary where you can enforce controls. And then I think we’ve all seen this in the news recently. There’s been a lot of stories around agents doing things that were unexpected and undesired. And one of the kind of, I guess, benefits of an agent is it’s given access to intelligence, it’s given a task, and often it can find unexpected ways to achieve that task. This is where we need guardrails and control. The sandbox provides a boundary where we can enforce those controls. We can be very explicit around whether that agent has access to the Internet, to other services, to MCPS, to APIs. We can enforce all of that at the sandbox layer. We can also observe at that boundary as well. We can see what is happening and confirm that it conforms with the policies. So think about the sandbox as it enforces isolation. It provides a boundary where you can apply controls and where you can apply observer observability. You can collect telemetry to see what’s happening. So it’s a really important building block for any agentic platform.
Swapnil Bhartiya: Your launch catalog runs from 8 billion to 2.8 trillion parameter models. You have added Blackwell ultra clusters with 288 GB of HBM3 memory per GPU. And you are preparing for Vera Rubin next year? Where do you tell a customer to take dedicated GPUs instead of serverless inference?
Jon Alexander: Yeah, this is a great point and I think a lot of this comes down to the individual customer, what their priorities are. And so the inference product that we’re launching, as you mentioned, or maybe take a step back, we’re expanding our fleet. So the types of GPUs that we have available, so we can support this broad diversity of models all the way from slm. So small language models that are only a few billion parameters, all the way up to these frontier level intelligence trillion plus parameter models, they have very different characteristics in terms of the amount of memory and compute that they require. And so we’re making sure that we’ve got the infrastructure, the types of GPU available that can support those. Now when we expose this into the inference platform, there’s several different paradigms that we can provide. So one is the shared serverless inference, where we abstract the inference and we just schedule the requests and we place those onto GPUs and then we destroy that after the request is complete. And then the next customer comes along and we schedule their work. But Akamai is in charge of scheduling the requests onto the GPUs. And then one of the key things we’re focused on is how do we make sure we get good utilization on that infrastructure. So that’s an optimization that we’re working on. That’s in the standard serverless inference where we’re running on top of shared infrastructure. With dedicated GPUs, we can give customers guaranteed capacity. We allocate a pool of capacity to those customers. They know that they’ve got guaranteed capacity always available. So this means that they’re not having to compete with other customers for resources. They’re not added to a queue, so they don’t have to wait. They can always know that they’ve got those resources available. They know that it’s only their workloads that are running on top of them. So it’s guaranteed performance. There’s no queuing and delays as they’re working on top of them. So for customers that have a good understanding of what their workloads look like, dedicated is often a way to get the performance the SLAs, the time to first token, the inter token latency that their businesses demand, and the ability to scale up to the peak capacity levels that they require if they know that there’s a certain tokens per second that they need to achieve. Dedicated GPUs are a good way of them achieving that.
Swapnil Bhartiya: Now, if you look at agent isolation, it is converging on several models right now. Micro VMs, policy and worse, containers and obviously webassembly isolated functions. How should developers choose between them for a given workload?
Jon Alexander: Yeah, it’s a really good question. And it’s complicated. I spent the weekend reading various launches from other customers. There’s now, I think, five different types of lambda runtime, different presentations of those serverless environments to support the different needs of agents. And I think it’s complicated. One of the things that we’re spending a lot of time thinking about is how do we make this easy for a developer? How do we enable that AI assisted developer experience for people building AI? How do we get to a point where it’s easy to select the right infrastructure for the right workload and not get tied in if your workload evolves be able to change. But right now I’d say there’s a couple of important characteristics that we’re looking at. And so what we’re launching here with our agentic runtimes, with sandboxes that we’ve just announced, they’re running on top of our webassembly platform that’s got a couple of really nice characteristics for ephemeral short lived workloads. So workloads that are going to run for milliseconds to handfuls of seconds, they spin up very, very quickly, they’re very light in terms of the resources that they’re consuming, so the cost is limited and then they get torn down afterwards. So you’ve got strong isolation separation, but very nice performance characteristics and a good cost benefit. The other kind of popular types of agent runtimes often leverage either micro VN or containers, things like gvisor or cater containers, those are much heavier weight and so they are slower to spin up, they consume more resources, they’re more expensive, but they do give access to the kind of a full computing environment. So a full file system, you can run a computer or browser use in there. So you can get access to a full computer, you can run a browser in there, you can do headless browser tasks and they can run for much longer durations, hours, days, weeks if you wanted. So you can have a much longer lived duration. So depending upon the modality, the type of agent that you’re looking at, you’re really picking one of these types of presentation. And our goal is to launch a container backed agent runtime in the first half of 2027. That’s something our team’s working on right now and adding the stateful elements to back that. So we’ll be opening up that in the next couple of months as we get into 2027.
Swapnil Bhartiya: You also plan to add capabilities that govern what agents can reach. And what can reach agents. What should teams building agent applications right now expect next?
Jon Alexander: We talked about how the agent sandbox provides this boundary, this isolation. It’s a control point. So you have the ability to control, enforce access, to define policies. That’s the first step. And so it provides that kind of infrastructure where you can add the controls. One of the next areas that’s critical and our team’s working on is providing that observability and telemetry to understand what’s going on, to ingest all the signals that are coming out of that sandbox so you can identify is it expected behavior or not. And one of the challenges that a lot of our customers are dealing with here is these are short lived milliseconds, seconds, and there’s often millions of them, millions of users interacting with the platform, each with their individual sandboxes that are being created and destroyed very, very rapidly. Inspecting these manually by human is impossible. It’s really, really difficult to ingest the volumes of data and to have these signals be interpreted by a human. So this is where we have to use AI and intelligence to inspect all of this data being generated and to derive insights from that and determine where we want to apply controls. So agent security is a key area that we’re looking at. We’re looking at kind of how do we ingest all the data, how do we develop the right threat models to identify the insights that need to be surfaced to a user or that need to be acted on? Where do we need to apply change? Where do we need to destroy a sandbox? Where do we need to alert a human to take intervention? So these two products are going to go hand in hand and really work together and so definitely encourage that sort of defensive depth or layered approach. Don’t just rely on the sandbox, work with a set of kind of restrictive controls. That’s a good starting point. But observing that feeding that data into an intelligence engine that can derive insights and learn from what’s going on and see if there is something unexpected happening is going to be really critical.
Swapnil Bhartiya: You also mentioned that enterprises should not have to choose between speed and control, which is quite the right approach. Where does that trade off show up most sharply in what your customers are asking for?
Jon Alexander: It’s a good point. I mean, I think today, today we are forcing developers to make these choices and there’s some tough trade offs that people are having to make. I think part of this is because there’s a lot of fragmentation across the industry. NeoClouds are providing the GPU infrastructure. We’ve got different CPU providers providing the agent runtimes. Then we’ve got software stack running on top of that for agentic harnesses, observability solutions, gateways. These are all different vendors and so they don’t have good integration, they don’t work seamlessly together. And I think that’s one of the opportunities that we’re looking at. How do we provide a better integration between these components? Our goal is to make this modular though. We’re not trying to require everyone to have an entire vertical stack from Akamai, we’re trying to make sure that you can integrate across. So say your GPUs are running somewhere else you can run an agent on Akamai. If you’re running the agent on Akamai how do you bring your own gateway if you’ve already got one? How do you route to Akamai as well as other platforms? If you’re in a multi cloud hybrid environment how do you get seamless observability over everything you’re doing? So we’re looking at how do we ingest all of those signals into one common platform and have one set of threat models that are looking across all of the data, all of the runtimes, all of the infrastructure. Only when we kind of have that ability to connect the different pieces together can we really get the true value.
Swapnil Bhartiya: John thank you so much for joining me and share these insights and I look forward to another great conversation. Thank you.
Jon Alexander: Great. Thank you very much. Great to be here.
Swapnil Bhartiya: And for those who are watching please go and check akamai.com to learn more about what they are doing in the Agent DKI space. Thanks for watching.
