By Most enterprises get AI by buying a full stack from a lab or a managed provider, and the bill climbs with every new use case. The same setup sends private business data to a third-party service and limits how much of the runtime you can tune. Cost, data control and the pace of innovation are where hosted AI starts to break.
In this interview on TFiR, Jon Alexander, SVP of Products at Akamai, breaks down the three challenges enterprises face with hosted AI: runaway cost, control over private data, and the pace of innovation.
Guest: Jon Alexander, SVP Products at Akamai
Show: The Agentic Enterprise
LinkedIn: Jon Alexander
Watch the discussion on YouTube
Here is what every platform engineer running AI workloads needs to know.
Technical Deep Dive
Q: What limits do enterprises hit with hosted AI services?
Jon Alexander, SVP Products at Akamai, says enterprises that buy a full AI stack from an AI lab or a managed infrastructure provider run into three main challenges: runaway cost, loss of control over private data, and limited ability to innovate. Alexander says these are the three things customers kept raising, and they led Akamai to build its own products.
“3 main challenges that we see our customers facing, which led to us investing in these products ourselves.” Jon Alexander, SVP Products, Akamai
Q: Why is enterprise AI deployment still limited?
Alexander says most enterprise customers are heavy consumers of AI but constrained in their use cases. They use it as a coding assistant, in product experiences such as natural language search, or in recommendations. Part of the reason is that building and deploying the full stack yourself is complex, so they typically buy from an AI lab or a managed infrastructure player, which is the easiest way to get access to frontier intelligence.
“Most of our customers are heavy consumers of AI, but they are constrained in the use cases they can apply it to.” Jon Alexander, SVP Products, Akamai
Q: How can enterprises control the cost of AI inference?
Alexander says costs are running away as adoption, usage and use cases grow across the enterprise, so customers are looking for ways to manage and control what they spend on AI. One route is moving from closed-source models to open source models. That gives near frontier-level intelligence at a much lower per-token or per-unit cost.
“Moving from closed-source models to open-source models is one way to access near-frontier-level intelligence at a much lower per-token or per-unit cost.” Jon Alexander, SVP Products, Akamai
Q: Why does sending private data to a third-party AI service create risk?
Alexander explains that many inference workloads only produce useful results when the model can see private enterprise information, the core business intelligence. With that context it can decide, for example, whether to recommend product A or product B. That data often includes heavily regulated personally identifiable information, and sending it to a third-party cloud service means losing some of those restrictions. Customers therefore want more control over their data, including localization and where it resides.
“That data is incredibly important to your business. It’s also data that you often want to keep under tight control.” Jon Alexander, SVP Products, Akamai
Q: Why does a hosted AI service limit innovation?
Alexander points to the pace of innovation as the third challenge. Teams want to control the full stack and ship product features and capabilities at their own pace. On a hosted third-party service they cannot fine-tune all of the runtime settings, which caps how much they can innovate.
“On a hosted third-party service where you don’t have the ability to fine-tune all the runtime settings, you’re limited in how much innovation you can really drive.” Jon Alexander, SVP Products, Akamai
Q: Why are enterprises focusing on inference instead of training?
Alexander says the industry moved its focus from training very large language models to deploying inference, which is where intelligence is used in production to drive business outcomes. Akamai made the decision 18 months ago to start investing in its AI stack based on that shift.
“Inference is where we’re deploying that intelligence and using it in production to drive business outcomes.” Jon Alexander, SVP Products, Akamai
Resources & Documentation
- Akamai Inference, Akamai’s inference product
- Akamai Agent Sandbox, Akamai’s sandbox product for AI agents
- Akamai Inference Cloud Platform, Akamai’s cloud platform for inference
***
👇 Click to Read Full Raw Transcript
Swapnil Bhartiya: If you look at Akamai and based on our conversations, Akamai’s AI strategy was built on the thesis that inference, not training, would take the majority of enterprise focus and investment. And that turned out to be true. Your customers proved that thesis and pulled you further. Can you talk about what were they still assembling themselves and why you’re adding managed inference and a sandbox now?
Jon Alexander: Yeah, exactly. No, it’s a, That’s exactly right. So it was 18 months ago from now that we actually kind of made the decision to start investing into building out Akamai’s AI stack. And as you said, we saw this transition happening where the industry was moving focus from training these very large language models into deploying inference. And Inference is where we’re deploying that intelligence, we’re using it in production to drive business outcomes. And this is kind of what we’re seeing with customers now. Most of our customers are heavy consumers of AI, but constrained it in those use cases. So when we talk to our enterprise customers, the people who are buying our CDN and security services today, they’re using AI probably as a coding assistant. They might be using it in some of their product experiences, for things like natural language search. Maybe they’re using it in some of their recommendations, but it’s limited kind of deployment. And part of that is because it’s complex to build and deploy this all yourselves. And so they’re typically purchasing from one of the AI labs, from one of these managed infrastructure players who’s giving them the full stack. And that’s the easiest way to get started and to get access to frontier intelligence. So that’s where we see most of the, industry is at today. But there’s a couple of challenges with that. So 3 main challenges that we see our customers facing, which led to us investing in these products ourselves. First one, probably no surprises around cost. This is not an insignificant investment that many of our customers are facing, and these costs are really running away as they’re driving more adoption and more usage, more use cases within their enterprises. And so customers are starting to look at ways of managing that cost and to control how much they’re spending on AI. So moving from closed-sourced models to open source is one way of getting access to near frontier-level intelligence at a much lower per-token cost or per-unit cost. So that’s the first piece. The second piece is around data. A lot of the inference workloads that we’re seeing today only drive useful results if you pass the model access to your private, secret enterprise information, your core business intelligence. You need to pass all of that context to the model so it can make the right decisions, make a decision whether to recommend product A versus product B, based on all the information you have around the user, around the insights you’ve gathered doing your business. That data is incredibly important to your business. It’s also often data that you want to keep close control over. This is personally identifiable information that is heavily regulated and that you need to control, and sending that off to a third-party service in the cloud means that you’re losing some of those restrictions. So the second piece that we’re seeing is customers wanna take more control over their data, wanna be able to retain the localization and where that data resides. The third piece then is really around innovation, the pace of innovation, the ability to control the full stack and to innovate on the product features and capabilities at the pace they want to. And again, on a hosted third-party service where you don’t have ability to fine-tune all of the runtime settings, you’re limited in how much innovation you can really drive. And so these were the 3 things we were hearing from our customers, and that’s what we set out to solve with these 2 new launches that we’re talking about.
