By Using a large language model to evaluate and guardrail every AI agent action in an enterprise environment is not a viable strategy. The cost scales with agent volume and quickly reaches tens of millions of dollars annually. Latency adds seconds to every interaction, making real-time control impossible. And because telemetry data must leave the organization to reach a frontier model, data sovereignty and compliance requirements become immediate blockers.
In this interview on TFiR, Kamal Hathi, SVP and GM of the Splunk Business Unit at Cisco, walks through how Splunk’s Luna small language model delivers 100% agent evaluation coverage at 97% lower cost than LLM-based approaches, running on-premises and fine-tuned on customer telemetry data.
Guest: Kamal Hathi, SVP and GM of the Splunk Business Unit at Cisco
Show: TFiR
Here is what every platform engineer and AI ops team needs to know.
Technical Deep Dive
Q: Why can’t enterprises use large language models to evaluate AI agent actions?
Kamal Hathi, SVP and GM of the Splunk Business Unit at Cisco, identifies three compounding failures with LLM-based agent evaluation. First, telemetry data sent to a frontier model leaves the organization’s perimeter entirely, creating data sovereignty and compliance risk. Second, using an LLM as a judge for every agent action is prohibitively expensive: evaluation costs scale directly with agent volume and quickly reach tens of millions of dollars. Third, LLM inference adds significant latency to every interaction, making real-time agent control impractical.
“Using this LLM large language model as a judge on your agent is extremely expensive. If every single action of your agent gets evaluated, now you’re paying like crazy. And latency is a problem. These things are not fast to evaluate.”
Kamal Hathi, SVP and GM of the Splunk Business Unit, Cisco
Q: What is 100% agent evaluation and why does sampling fall short?
Because LLM-based evaluation is slow and expensive, most teams resort to sampling: evaluating only a fraction of agent actions rather than all of them. Hathi argues this defeats the purpose of guardrailing, since unsampled actions are uncontrolled by definition. Splunk’s approach with Luna is to evaluate every single instrumented agent action against a predefined set of criteria, called evals, that operators configure before an agent goes into production. If an action meets the criteria, it is allowed; if not, it is blocked.
“Our approach is to give you 100% sampling, 100% evaluation. It’s 97% cheaper than using a large language model and it’s much, much faster.”
Kamal Hathi, SVP and GM of the Splunk Business Unit, Cisco
Q: What is Splunk’s Luna small language model and what does it actually do?
Luna is a small language model built specifically for agent observability inside Splunk’s agent evaluation harness. Its sole function is to assess whether an agent action meets the evaluation criteria an operator has defined, then allow or block the action accordingly. Because it is purpose-built for this narrow task rather than general reasoning, it runs far faster and far cheaper than a frontier LLM, and it does not require external API calls, keeping all data on-premises.
“Luna is a small language model that’s part of our agent observability solution. Its role is basically every time you should use it when you start to build these autonomous or AI-based solutions in your organization.”
Kamal Hathi, SVP and GM of the Splunk Business Unit, Cisco
Q: How does fine-tuning Luna on customer telemetry data reduce hallucinations?
Generic large language models have no knowledge of an organization’s specific infrastructure, tooling, or operational patterns, which is a primary driver of hallucination in agent evaluation. Hathi explains that Splunk allows customers to fine-tune Luna using their own telemetry data, making the model’s evaluations specific to that environment. The combination of a narrow task scope and domain-specific fine-tuning reduces hallucination in evaluation outputs to near zero compared to a general-purpose frontier model.
“You can fine tune it with your own data, your own telemetry. So now it’s extremely focused, it’s extremely tuned. So the hallucination is almost non-existent versus something that’s so generic where it’s trying to understand this in the context of everything else.”
Kamal Hathi, SVP and GM of the Splunk Business Unit, Cisco
Q: What does the cost reduction from LLM-based to Luna-based agent evaluation look like in practice?
Hathi describes a customer case where evaluation costs dropped from tens of millions of dollars annually using a large language model to approximately $100,000 with Luna for the same evaluation workload. He frames the broader math: if an organization wants to run 20 agents and LLM-based guardrailing costs $20 million per agent, the total evaluation budget reaches into the billions, making agentic deployment economically impossible. Luna brings that to a range enterprises can actually budget for.
“They were spending literally on the order of tens of millions of dollars on large language model evaluation. And the same thing now with our model is costing maybe $100,000. It suddenly goes from impossible to practical.”
Kamal Hathi, SVP and GM of the Splunk Business Unit, Cisco
Q: How does Agent Launchpad differ from Luna and agent observability?
Hathi draws a clear boundary between the two products. Agent Launchpad is focused on agent creation and setup, providing the tooling to build and configure agents, including Splunk’s own internal agents such as the Agentic SOC for security operations. Luna and the agent observability solution sit downstream of that: once agents are built and running, agent observability handles the ongoing evaluation, guardrailing, and control of every action those agents take. The two are complementary but serve distinct phases of the agent lifecycle.
“Agent Launchpad is more a matter of setting up, creating agents. Luna is used in the context of agent observability, which is the harness that uses this model to do the actual evaluations, to do the guardrailing, to do the actual actions.”
Kamal Hathi, SVP and GM of the Splunk Business Unit, Cisco
Q: Who owns agent observability in the enterprise: the security team or the platform team?
Hathi argues the question itself reflects an outdated org structure. With deterministic software, there was a clear boundary between platform engineering (which instruments and observes code) and security (which hunts for exploits). With AI agents, non-deterministic behavior means a hallucinating agent and a compromised agent can produce the same observable outcome. As a result, he is seeing CISO and CIO teams converge on agent observability tooling together, rather than operating in separate silos that only intersect during incidents.
“With the nature of agentic solutions, this kind of non-deterministic behavior, the line between intentional and unintentional starts to bleed. The CISO team and the CIO team are increasingly collaborating on these kinds of things.”
Kamal Hathi, SVP and GM of the Splunk Business Unit, Cisco
Q: Does Splunk use Luna to observe its own internal agents?
Hathi confirms that Splunk runs its own internal agents, including the Agentic SOC for security detection and remediation, through the same Luna-based agent observability harness it sells to customers. He describes this as a recursive pattern: the agent observability system monitors the agents that were built to monitor other things. This means Splunk’s production claims about Luna’s performance are validated against its own internal workloads, not only customer deployments.
“Our own agents, we observe with our own agents. It’s like this recursive thing.”
Kamal Hathi, SVP and GM of the Splunk Business Unit, Cisco
Resources & Documentation
- Splunk (Cisco), agent observability, Luna small language model, and the Agentic SOC platform for enterprise AI evaluation and security operations
***
👇 Click to Read Full Raw Transcript
Swapnil Bhartiya: Of course, when it comes to general purpose data that is used by, you know, to train. Because I mean, if you look at LLM, we talked about the term AI, but it’s basically LLMs, you know, that has vast amount of data. But when it comes to telemetry data that could be very, very specific to a specific industry specific company specific set of tools that you’re running. So first of all, it’s not going to help that pool of knowledge base. And second is that data is also leaving the organizations.
Kamal Hathi: But third thing is that most of these models, they don’t know how to deal with. They have no idea.
Swapnil Bhartiya: Exactly. So since they don’t have any data, they cannot even give you right response to that as well. So talk a bit about what kind of challenges create for organizations because they do want to use AI, they do want to use agents to act upon it. Because that telemetry data can be for some outage happen. I mean there are ton of things because that’s a. So talk a bit about what approach either Cisco or Splunk is taking. You mentioned we get offset with the large language models. But the fact is in most cases you need a small. That’s right, very focused. Yeah. So let’s talk about how Splunk and Cisco are approaching this.
Kamal Hathi: Yeah, let me give you specific examples. So we talked a lot about this notion of agent observability. This is this idea that agents basically you cannot assume that they’re going to do what you ask them to do. This is just a fallacy. They have to be evaluated, they have to be controlled, they have to be guardrail. So for that you need to be able to observe and evaluate every agent action. Literally you go in and you set up what I call these evals upfront before the agent goes into production. So you do this. Now the problem with that is that a what you described before, which is do you want every agent action going to some kind of large. This is just. You are giving away everything. But the second and third important parts are that a using this LLM large language model as a judge on your agent is extremely expensive. If every single action of your agent gets now you’re paying like crazy. The second thing is, third thing is that latency is a problem. These things are not fast to evaluate. So the approach we take is we have specific small language model, it’s called the lunar language model that we use for these agentic evals. What we do is we do 100% evaluation. Every single agent action that is instrumented gets evaluated and it’s compared against what you had intended to do and then based upon that, then you are either allowing the agent to take the action or you block it. Now this is very important because if you do the other approach, which is you take the large language model, companies end up sampling it, they sample everything. The latency is very high and as a result you don’t really get what you want. So our approach is to give you 100% sampling, 100% evaluation. It’s 97% cheaper than using a large language model and it’s much, much faster. So because it’s bespoke, because it’s built to do this simple set of activities, and then we allow you to take your telemetry data and fine tune this model so that when we do, the evaluation is based upon your data and it runs all on your premises, so it’s all under your control and it’s cheap, it’s fast, and it’s highly governed. So this is really important and that’s the approach we take. And I think trying to use an LLM for that, for example, has lots of reasons why it won’t work.
Swapnil Bhartiya: So when we look at Luna, so is it more or less like a harness to get us started? Is this kind of. And you also, agent Launchpad also, you folks launch today, so talk a bit about what exactly it is and at what point its role starts and when it ends or it is all across because you continue to fine tune model and it’s best spoke for your specific.
Kamal Hathi: So Luna is a small language model that’s part of our agent observability solution. And what that solution is, it’s all about setting up evaluations and guardrails for your agentic solution that you’re building internally or third parties that you have got. And so this is available now and its role is basically every time you should use it. When you start to build these autonomous or AI based solutions in your organization, that’s its role. It is not something that’s going to do everything, but it’s very focused on this one thing. Is it different than Agent Launchpad, which you mentioned? But that’s more matter of setting up, creating agents. For example, our own internal agents, we have this thing for security we call the Agentic soc, which is a security operations center and that’s an agent that does look at security tasks for detection or remediation, etc. We also run those in the context of our agent observability. Our own agents, we observe with our own agents. It’s like this recursive thing. And so the role of this, what I just described, the lunar model that it’s used in the context of agent observability, which is the harness that uses this model to do the actual evaluations, to do the guard railing, to do the actual actions, et cetera. But it’s a model, the model that runs in the context of the harness, which is the agent observability solution. And that in turn is what is being used by customers and by ourselves to make sure that all the agents that they’ve got in the enterprise are being effectively evaluated and controlled.
Swapnil Bhartiya: And those agents can be totally using large language models, anything they want. So once you have this foundation in place with Luna, how do you help team actually scale without once again going to the whole sinkhole of cost and hallucination and performance?
Kamal Hathi: That’s the point. So the important part here is that this model is very much geared towards one thing only, which is to make sure that you are able to take a set of criteria you define, what’s called an evaluation eval. And making sure that your agent is meeting those criteria, that’s what it’s geared for. And because of that, its cost is much less. I said 97% cheaper than using a large language model. And so an example, I can’t give you the company’s name, et cetera, but they were spending literally order of tens of millions of dollars on large language model. And the same thing now with our model is costing maybe $100,000. I’m just giving you orders of magnitude. But it suddenly goes from if you’re a company who wants to have 20 agents and each agent evaluation guardrail with large language model cost $20 million, you want to spend billions of dollars on that. Whereas with our solution it becomes practical. It’s not cheap cheap, but it’s way cheaper than that. So I think it’s really important to understand that our goal and what we are making, delivering with this is practicality and also it’s about speed. If it takes seconds of time at every interaction to do this, then it’s useless. We make it very, very fast, very, very cheap. And again, it’s very focused. Hallucinations also become interesting here because now this model is very focused, but also you can fine tune it with your own data, your own telemetry. So now it’s extremely focused, it’s extremely tuned. So the hallucination is almost non existent versus something that’s so generic where it’s trying to understand this in the context of everything else. So that’s the power of using focus models and using and having them tuned with your data. And we have other examples as well. For example, we talked today, if you noticed or not, but about a time series model. That time series model is based upon again we deliver to our customers, they can fine tune with their own data and then it tries to start predicting anomalies in a time series of events. But again, it’s focused, it’s small, it does not depend upon some large frontier model. And so it becomes much, much more effective much faster and something that can be used in enterprise context.
Swapnil Bhartiya: And if you look at Luna, who is its target audience? Who is its buyer? Is it security teams? Of course, data teams or. I think we’ve tried to break those silos back in those days, but now we have created new silos where teams don’t even talk to each other. It’s only when some incident happened, then everybody comes together. So talk about, you know, or maybe Luna can become a catalyst to bring two teams together.
Kamal Hathi: I think it’s not necessarily a matter of whether or not our model does it, but the fact remains agents are becoming like a new app, right when that happens, let’s take an example. Let’s say you have a customer service agent whose job is to answer questions, do refunds, et cetera. And that agent has a hallucination, a problem, and it starts returning instead of $10, $100, so 10 times what it should. Now imagine the same agent that has been hacked and it returns $100 instead of $10. What’s the difference? It’s the same thing, the same problem, Security, observability. In the past you would had a hard line because this was a matter of writing code, observing, instrumenting it. This was all about having exploits and all that with the nature of solutions, with agent and AI and agentix solutions, this kind of non-deterministic behavior, the line between intentional and unintentional starts to bleed. So who is it for? Is an interesting question to answer. Now what we are seeing is that the CISO team and the CIO team are increasingly collaborating on these kinds of things. The CIO is geared to building agents that give value and solution. The CISO is worried about these agents starting to do malicious act. And so both of them are increasingly using this as a way to combine and converge. And we are seeing this more and more. In fact, to your point about in the past, when there was a problem or incident, only when they start to intersect, then they would start to talk. There’d be silos otherwise. So the new world. These solutions are increasingly converged already, and troubleshooting an agent or looking for security is very, very.
