AI Infrastructure

How to Switch Inference Providers With AI Gateway Routing | Jon Alexander, Akamai | TFiR

0

Moving a production AI application to a new inference provider sounds like a rewrite. It does not have to be. Still, once traffic can go to more than one provider, someone has to decide which model serves which request, and what happens when a user’s budget runs out.

In this interview on TFiR, Jon Alexander, SVP Products at Akamai, walks through what changes on day one for a team with an application in production, how gateway routing works, and why agents need a sandbox boundary.

Guest: Jon Alexander, SVP Products at Akamai
Show: The Agentic Enterprise
LinkedIn: Jon Alexander
Watch the discussion on YouTube

Here is what every platform engineer needs to know.

Technical Deep Dive

Q: How do you switch an AI application to a new inference provider without rewriting it?

Jon Alexander, SVP Products at Akamai, explains that because Akamai Inference uses an OpenAI-compatible API, a team with an application in production changes one line of code on day one. Instead of pointing at another service, the team sets that line to the Akamai API endpoint and adds the token key from Akamai, and requests route to Akamai immediately. Alexander says this makes it very easy to get started.

“It’s just one line of code that needs to change to start using the Akamai Inference Platform.” Jon Alexander, SVP Products, Akamai

Q: How does AI gateway routing decide which model handles each request?

Alexander says teams can add routing logic beyond the single endpoint change. Routing can work at the model level, sending requests for Model X to Akamai and Model Y to another provider. It can also work per user: if someone has exceeded the monthly budget, a gateway policy can route them to a medium-sized, lower per-token-cost model on a different provider instead of the frontier-level model. The gateway also gives teams observability and visibility into where requests go, so routing decisions can be made at a fine-grained level.

“That AI Gateway is a great way to manage which inference platform you’re sending requests to and add this intelligence.” Jon Alexander, SVP Products, Akamai

Q: Why do AI agents need a sandbox boundary?

Alexander points out that many agents are ephemeral: they spin up, do some work, and spin down. They must be isolated from each other so one agent cannot access another customer’s data. An agent is also given intelligence and a task, and it can find unexpected ways to complete it, which is why guardrails matter. The Akamai Agent Sandbox provides a boundary where controls are enforced and telemetry is collected, so teams can confirm the agent conforms to policy.

“We can be very explicit about whether that agent has access to the internet, other services, MCPs, or APIs. We can enforce all of that at the sandbox layer.” Jon Alexander, SVP Products, Akamai

Q: Where does the WebAssembly technology behind Akamai’s agent sandbox come from?

Alexander says Akamai acquired Fermyon, a company that provided WebAssembly functions. Akamai originally used them to deploy functions at the edge of its platform for personalization, recommendations and transformations of content flowing through its CDN network. They also extended Akamai’s security controls for bot detection and for authentication and authorization logic. Alexander sees that same core technology as a foundation for an agentic runtime.

“But that core technology provides a great foundation for something like an agentic runtime.” Jon Alexander, SVP Products, Akamai

Resources & Documentation

  • Akamai Inference, Akamai’s inference service for open-weight and custom models through OpenAI-compatible APIs
  • Akamai Agent Sandbox, a WebAssembly-based boundary between an agent and the code it generates

***

👇 Click to Read Full Raw Transcript

Swapnil Bhartiya: As you mentioned, Akamai Inference serves open-weight and custom models through OpenAI-compatible APIs. So existing applications move without rewrites. For a team that’s already working on it, that already has an application in production, what actually changes on day 1?

Jon Alexander: Yeah, this is one of the interesting things. So with, we mentioned the OpenAI-compatible API interface. So one of the nice things is It’s one line of code that has to change to switch to start using the Akamai inference platform. And so rather than pointing at another service, change that one line to point to the Akamai API endpoint, add the token key from Akamai, and you’re immediately routing your token request to Akamai versus another platform. Makes it very, very easy to get started. You can add more sophistication around how you route tokens. And so you can do model-level routing. So you can say, hey, if a user is using Model X, I’m going to send that to Akamai. Model Y, I’m going to send that to another provider. You can even add per-user logic. So maybe I’ve exceeded my budget for the month. So in the gateway, you can set a policy where you can route me to another model on a different provider because I’ve exceeded my budget. So I don’t get access to that frontier-level model anymore. I’m going to get access to a medium-sized model where it’s lower per-token cost. So there’s a lot of intelligence that you can add. That AI Gateway is one of the great ways to manage which inference platform you are sending requests to and to add this intelligence. So this is one of the things that we encourage our customers to have control. Gives you observability and visibility. You can see what’s going on, but then you can make these routing decisions. You can add intelligence to decide whether to send to provider A, provider B, and to apply those routing decisions at very granular, fine-grained levels.

Swapnil Bhartiya: As you explained, Agent Sandbox puts a WebAssembly boundary between an agent and the code it generates. separated even from the agent’s own runtime. Why does that separation matter and why ship it as an open source component of Spin rather than keep it proprietary?

Jon Alexander: Yeah, well, I mean, this goes back to one of the things that we talked about, Swapnil, maybe a year ago. It was back in December, Akamai acquired a company called Fermyon who provided WebAssembly functions that originally we were using to deploy functions on the edge of our platform for things like personalization, recommendations, doing transformations, for content as it was flowing through our CDN network. It allowed us to extend our security controls for things like bot detection and additional logic around authentication and authorization. But that core technology is the great foundation for something like an agentic runtime. And so when we think about an agent, a lot of these are ephemeral, short-lived. They need to spin up, do some work, and then spin down. You want these to be isolated from each other. You don’t want one agent to be able to access the data of another customer. So having isolation between the agents is important. So this is the first thing that the sandbox provides. It isolates what that agent can get access to. It provides that boundary where you can enforce controls. And then I think we’ve all seen this in the news recently. There’s been a lot of stories around agents doing things that were unexpected and undesired. And one of the kind of, I guess, benefits of an agent is it’s given access to intelligence. It’s given a task, and often it can find unexpected ways to achieve that task. This is where we need guardrails and control. The sandbox provides a boundary where we can enforce those controls. We can be very explicit around whether that agent has access to the internet, to other services, to MCPs, to APIs. We can enforce all of that at the sandbox layer. We can also observe at that boundary as well. We can see what is happening and confirm that it conforms with the policies. So think about the sandbox as it It enforces isolation. It provides a boundary where you can apply controls and where you can apply observability. You can collect telemetry to see what’s happening. So it’s a really important building block for any agentic platform.

How to Choose Workload Placement in the AI Era | Bryan Litchford, Rackspace Technology | TFiR

Previous article