By Centralized AI infrastructure works until it does not. When AI agents move from answering human queries to triggering autonomous machine actions, the latency window shrinks to a point where centralized compute cannot reliably meet it. Choosing the wrong inference architecture for the wrong workload creates failure conditions that show up in production, not in planning.
In this interview on TFiR, Jon Alexander, SVP of Product for the Cloud Technology Group at Akamai, walks through how to segment AI inference use cases by latency sensitivity and when distributed inferencing becomes an operational requirement rather than an optional optimization.
Guest: Jon Alexander, SVP of Product for the Cloud Technology Group at Akamai
Show: TFiR
Here is what every platform engineer and AI infrastructure architect needs to know.
Technical Deep Dive
Q: Does distributed AI inferencing apply across all industries or only regulated and latency-sensitive ones?
Jon Alexander, SVP of Product for the Cloud Technology Group at Akamai, argues that distributed AI inferencing will have broad applicability well beyond regulated or traditionally latency-sensitive industries. The determining factor is not the vertical but the nature of the workload: whether a human or a machine is waiting on the output and what the cost of delay is in that specific context. Workloads with high latency tolerance, such as batch processing jobs, remain well-suited for centralized infrastructure regardless of industry.
“I think this will have broad applicability.” — Jon Alexander, SVP of Product for the Cloud Technology Group, Akamai
Q: Which AI inference workloads can safely run on centralized infrastructure?
Alexander identifies batch processing as the clearest candidate for centralized infrastructure. These workloads are not time-sensitive, so the round-trip latency of sending data to a central data center and receiving a result back does not create a functional problem. The key criterion is whether the workload can tolerate the additional milliseconds that centralized routing introduces without degrading the user experience or the system outcome.
“Not every AI inference use case needs to be real time or low latency. Things like batch processing are not time sensitive, so those can be run in centralized infrastructure.” — Jon Alexander, SVP of Product for the Cloud Technology Group, Akamai
Q: Why do human-in-the-loop AI applications require low-latency distributed inference?
When a human is waiting for a response from an AI system, latency is bounded by human perception and patience. Alexander frames human tolerance as finite, meaning delays that exceed a perceptible threshold degrade the experience and reduce the practical utility of the application. Any AI workload where a user is actively waiting on output falls into the performance-sensitive category and benefits from inference being executed closer to the source of the request.
“Anything that a user is in the loop, like a user waiting for a response, human tolerance is finite.” — Jon Alexander, SVP of Product for the Cloud Technology Group, Akamai
Q: Why are machine-to-machine AI workflows more latency-sensitive than human-facing ones?
Alexander draws a direct contrast between human and machine latency tolerance, stating that machine tolerance will be even more sensitive than human tolerance. When an AI agent produces output that triggers an automated downstream action, any delay in inference propagates immediately into system behavior. There is no human buffer to absorb or compensate for the lag, making the latency requirement more precise and less forgiving than in human-facing applications.
“If we are thinking about machines needing to take action based on the output of an agent, that is where we are going to find very, very sensitive latency targets to be achieved.” — Jon Alexander, SVP of Product for the Cloud Technology Group, Akamai
Q: How should teams segment AI inference use cases to decide between distributed and centralized deployment?
Alexander’s framework segments workloads along a single axis: latency sensitivity. Batch and asynchronous workloads with no user or machine waiting on immediate output belong in centralized infrastructure. Performance-sensitive workloads, specifically those with a human waiting on a response or a machine that must act on an agent’s output, require distributed inference to meet their latency targets. The decision is architectural and should be made per workload type, not applied uniformly across a platform.
“A large number of use cases are performance sensitive. Not everything needs to be distributed, but where performance matters, it really matters.” — Jon Alexander, SVP of Product for the Cloud Technology Group, Akamai
Resources & Documentation
- Akamai, cloud computing, edge delivery, and AI inferencing infrastructure platform
***
👇 Click to Read Full Raw Transcript
Swapnil Bhartiya: Is this distributed approach specific to certain industries, highly regulated, real time or latency sensitive ones? Or should it become a standard model for AI inferencing across the board?
Jon Alexander: Yeah, I think this will have broad applicability. And so not every AI inference use case needs to be real time or low latency. It doesn’t need that performance. So there are certainly applications you can think of that will have a high tolerance to latency. Certainly things like batch processing, those aren’t time sensitive, so those can be run in centralized infrastructure. And so not everything needs to be distributed. But a large number of use cases are performance sensitive. And certainly as you’re thinking about anything that a user is in the loop, like a user’s waiting for a response, human tolerance is finite. Machine tolerance is going to be even more sensitive. Again, if we’re thinking about machines needing to take action based on the output of an agent, that’s where I think we’re going to see. Very low tolerance for latency. That’s where I think we’re going to find that it’s going to require very, very sensitive latency targets to be achieved.
