By Most AI agent deployments start with the wrong architectural assumption. Teams treat the LLM as the agent, then wonder why their infrastructure costs are unpredictable and their reliability is poor. The compute requirement for a production AI agent is not a GPU problem. It is a distributed systems problem spanning CPU workloads, storage layers, retrieval pipelines, and multiple specialized models running in coordination.
In this interview on TFiR, Dr. Robert Blumofe, Executive Vice President and Chief Technology Officer at Akamai, breaks down the actual architectural composition of AI agents, explains how to allocate functionality between AI and non-AI tools, and makes the case for right-sizing LLMs to specific tasks rather than defaulting to large general-purpose models.
Guest: Dr. Robert Blumofe, Executive Vice President and Chief Technology Officer at Akamai
Show: TFiR
Here is what every platform engineer and AI infrastructure architect needs to know.
Technical Deep Dive
Q: What is an AI agent architecturally, and why is it wrong to treat it as just a powerful LLM?
Dr. Robert Blumofe, Executive Vice President and Chief Technology Officer at Akamai, explains that an AI agent is a system composed of many components, not a single model. An LLM plays a central role in managing natural language interaction and decision-making about how the interaction should proceed, but on its own an LLM can do nothing but produce text. What makes a system an agent is the ability to act, and that requires tools that translate text output into executable actions.
“An LLM can do nothing but produce text. If you want to do anything, you have to translate that text output into action. And that means tools.” — Dr. Robert Blumofe, Executive Vice President and Chief Technology Officer, Akamai
Q: What kinds of tools make up a production AI agent?
Blumofe describes a typical agent as a combination of multiple AI models plus a wide variety of non-AI tools. Examples he gives include tools for reading and writing storage, retrieving customer data from a CRM, fetching information from the web, querying a vector database for private information, sending email, managing calendar, and calculating shortest paths on a map. The diversity of these tools is what drives the hybrid infrastructure requirement.
“Most agents are a combination of AI models, multiple AI models, plus a whole variety of tools.” — Dr. Robert Blumofe, Executive Vice President and Chief Technology Officer, Akamai
Q: Should non-AI tools handle most of an agent’s functionality?
Blumofe offers a direct design principle: put as much functionality as possible into non-AI tools, and use AI only when nothing else will work. He argues that for tasks like email, database retrieval, and web search, dedicated non-AI tools are significantly more efficient and more reliable than routing those tasks through an AI model. AI should be reserved for what it uniquely enables, specifically natural language interaction and reasoning about how to proceed.
“A good rule of thumb if you are designing an agent is to put as much of the functionality as possible into the non-AI tools. Use AI only when nothing else will work.” — Dr. Robert Blumofe, Executive Vice President and Chief Technology Officer, Akamai
Q: Why do AI agents require hybrid infrastructure rather than just a GPU cluster?
Because agents are multi-component systems, their infrastructure demands are similarly distributed. GPUs are needed for LLM inference, but CPU is required to run algorithms like shortest path calculations and SQL queries. Storage is needed for memories and vector database retrieval. Blumofe concludes that the hybrid infrastructure requirement is a direct consequence of sound agent architecture, not an edge case.
“You do need GPUs, but you also need CPU to run that shortest path algorithm, to run the SQL query. And of course you need storage for all that data.” — Dr. Robert Blumofe, Executive Vice President and Chief Technology Officer, Akamai
Q: When should you use a smaller specialized LLM instead of a large general-purpose model?
Blumofe argues that not every agent use case requires a multi-trillion parameter general-purpose model. When an agent is built for a specific domain, a smaller LLM specialized for that task will typically perform better and cost less. He uses the example of an insurance claims agent: that system does not need the ability to write code, compose poetry, or answer trivia. Matching the model to the scope of the task is a core design discipline, not an optimization afterthought.
“If you are building an agent to help your customers file insurance claims, you probably do not need an agent that can write code, compose sonnets, tell jokes, and give you the cast of every MASH episode.” — Dr. Robert Blumofe, Executive Vice President and Chief Technology Officer, Akamai
Resources & Documentation
- Akamai, cloud computing, security, and content delivery platform referenced throughout the discussion
***
👇 Click to Read Full Raw Transcript
Swapnil Bhartiya: When we talk about AI agents, everyone immediately fixates on GPU scarcity. Can you explain why AI agents actually require a hybrid infrastructure rather than just a massive GPU cluster?
Dr. Robert Blumofe: You know, this is a great point and you know, I think the more, I think people can really wrap their heads around what an agent really is architecturally, the better off, better off will be because I think it’s tempting to think that, well, you know, an AI agent is simply a super powerful LLM with the latest and greatest LLM reasoning capabilities. That’s an agent and that’s not the case. The key insight probably is to think of an agent as being a system with many, many components. In fact, most agents do indeed have many, many components. And an LLM is just one of those components. The LLM. Typically, an LLM will typically play a fairly central role in the agent because you need something that’s going to manage the natural language interaction, you need something that’s going to make about how the interaction should proceed, what’s the next question to ask, what’s the next task to do, and so on. So an LLM or in many cases multiple LLMs play a fairly central role. But ultimately what makes these things agents is the ability to do things. And remember, an LLM can do nothing but produce text. If you want to do anything, you have to translate that text output into action. And that means tools. That’s the key thing.
Swapnil Bhartiya: Agents.
Dr. Robert Blumofe: Agents are systems that involve LLMs using tools. And in most good agents there’s typically quite a few tools. It could be tools to read and write, storage, it could be tools to retrieve, retrieve information from say a, a CRM, right, to retrieve information about the customer that you’re talking to. It may be a tool that retrieves information off of the web. It may be a tool that retrieves private information from a so called vector data database, could be a tool to send email, manage calendar, it could be a tool to calculate shortest paths on a map. So most agents are a combination of AI models, multiple AI models, plus a whole variety of tools. And I would argue, by the way, and I’ve been saying this for a while now, that a, a good rule of thumb if you’re designing an agent is to put as much of the functionality as possible into the non AI tools. You know, in some sense use AI only when nothing else will work, you know, and, and that doesn’t mean don’t use AI, of course, because the AI as the central component to making, making decisions and managing the natural language interaction well, nothing else will work. AI does that and it does it so well. But when it comes to other tasks, like things that I mentioned, like email retrieving things from a database, searching the web. No, use it, use an actual tool, a non AI tool. It’s way more efficient and way more reliable. So rule of thumb should be put as much functionality in your agent as you can into the, into the non AI tools. Okay? The upshot of all that is that the infrastructure demands are coming from not just the LLM itself, but from the combination of multiple LLMs, multiple tools, using data, retrieving data. So you have a hybrid infrastructure requirement. You do need GPUs, but you also need CPU to run that shortest path algorithm to run the SQL query and so on. And of course you need, you need storage for all that data that you’re going to be operating on, whether it’s storing things like memories or, or retrieving things from a vector database. So you have this hybrid need. And by the way, you touched on this earlier, you know, another point I would make about sort of a good design role is for the parts that are AI, the parts that are, say LLMs, use the right LLM for the job. Not everything requires a multi trillion parameter, ask me anything model. In many cases, if you’re building a, an agent for a specific use, you really can, you’re really going to be much better off with a, with an LLM that’s much smaller and specialized for that particular, for that particular task. If you’re building an agent to help your customers file insurance claims, you probably don’t need an agent that can write code, compose sonnets, tell jokes, and give you the cast of every MASH episode that ever was recorded. So, you know, use the right tool for the job and use the right AI for the job.
