How to Govern AI Agents Before They Go Wrong | Raj Koneru, Kore.ai | TFiR

Swapnil Bhartiya: Now, when it comes to AI today, enterprises are not struggling to build AI agents anymore. That problem is solved. The real struggle is to control the ones they have already built. Every team spins up its own agent and then nobody tracks what any of them can do or they are doing and who is actually accountable or responsible when things go wrong, when agents start acting on their own and bad things happen, that is already happening in production and companies are trying to solve their problem. Core AI has built its platform Artemis to solve exactly that. Governance first from day one. And today we have with us Raj Koneroo, founder and CEO of Kurai, to deep dive into this topic. First of all, Raj, it’s great to have you on the show.

Raj Koneru: Thank you. Thank you, Swapnil.

Swapnil Bhartiya: These days, almost everybody is using AI. I just came back from Splunk conference and the topic is same. You know, AI agents think they are there, powerful, but governance around them harness around. That’s a big topic. I want to hear from you. First of all, talk a bit about what in this space, either you saw. I mean, there are a lot of problems, so there’s not one problem you saw, but where you felt like, you know what, this is what we need to do. And that led to the beginning of this company. Talk about the story of the company.

Raj Koneru: Well, actually the story goes back about 12 years ago. So I saw how enterprises built and used applications right before AI, you know, going back into the Windows client applications, then the web applications and the mobile applications, and then game messaging apps. So the idea back then was, can we have an application which is headless, which doesn’t have a screen with buttons and fields and things like that, which can be deployed through natural language and which could interface and integrate with your backend systems to be able to provide answers and execute transactions for a consumer who uses an application. So we started Core AI with that mission that we can build a conversational AI platform on which conversational AI applications can be built. Now, going back, you know, 10, 11 years ago, you don’t. You did not have the big large language models, you had machine learning models. So therefore we had to build our own semantic models to do what used to be called NLP, Natural Language Processing, which is the understanding capability in natural language, and then be able to execute a workflow to satisfy the user’s needs. So that’s how we started. Obviously the models got better over a period of time and now we have LLMs. A number of the challenges that existed in terms of natural language understanding, entity collection, executing workflows, a lot of that has been enhanced and solved, but yet at the end of the day, it’s a chat application, either in a voice channel or in a digital channel. And today with the LLMs you try to take advantage of the LLMs doing some amount of reasoning and also LLMs being able to generate a contextually accurate response for a request, basically. So I think people focus a lot on the problems of deploying AI today and probably don’t talk enough about the opportunity. The opportunity for a user to communicate in natural language and conduct transactions is enormous. First of all, it’s natural to humans because we all speak basically and we communicate in language. So the opportunity to be able to bring efficiencies by people using these AI applications is immense. There’s efficiency gains, there’s cost reductions, the speed of business basically, which is enormous. And the knowledge that these LLMs already have embedded into them and the knowledge that’s fed to them as part of an application is also huge. So I think the whole world, every enterprise, everywhere in every corner of the world is now trying to figure out how to do this efficiently, basically. And that’s why we come in, we’ve learned how to do this really well. And it’s not just the governance. The governance is one piece of it. It’s being able to contextually execute a workflow basically to achieve a certain goal essentially. And that requires many components in it. And those are the components that we built into our platform to make it easy for enterprises to design, build, test, deploy, manage, measure and optimize these AI agents or multi agent systems.

Swapnil Bhartiya: Thank you so much. And you’re absolutely right about the whole LLMs. And actually AI has been around for so long when we talk about whatever super intelligent LLMs are, the one that are making these interactions possible, when you can ask questions, I mean, Siri and all those things were there, but suddenly things have changed. And also early days all these technologies were available only to data scientists and they were very expensive. Now it’s democratized where you have it on your phone and everybody is actually using and leveraging it. Now let’s talk about Artemis as well. What is it and how would you describe the platform’s architecture?

Raj Koneru: So Artemis is an enterprise AI platform so that enterprises can build any type of use case on it. Broadly speaking, there are three primary use cases that most enterprises build. The number one which many of the B2C enterprises build is customer service. Being able to service customers who call them on the phone, come on chat, make a request by email, whatever the interaction model is. And Artemis supports over 40 different such channels, basically. So automating customer service, providing very fast and efficient and accurate customer service represents the brand of that enterprise, basically, which leads to customer retention, customer loyalty, which leads to increased sales, and all stemming from satisfying customers. The second area is what we call AI for work, which is employees within an enterprise being able to use AI agents to do things more efficiently. If a task took a full day for an employee, now it can be done in 10 minutes, basically. So the productivity gains are just immense, essentially, which means employees can do more, which means the enterprise can grow faster, be more profitable, all of those things. And third one, which I think is actually one of the biggest opportunities is what we call AI for process, which is you have enterprises with these complex processes with a straight path and then variants to that straight path, basically handling different scenarios, whether it’s order to cash, procure to pay, supply chain management, whatever the case may be. So traditionally a workflow based process was very rigid, it was step by step and it was difficult to build and difficult to maintain. With agentic process automation you bring in intelligence basically that dynamically executes the process in a much more efficient way than before. So Artemis provides the tools and the frameworks to be able to build any one of these use cases. And it’s done using AI. So you design with AI, you build with AI, you evaluate with evals that are built on AI, you use guardrails that are embedded with AI, and then finally you deploy with AI, you measure with AI. And most importantly, I believe going forward is optimize what you have built with AI. So Artemis was built by itself using AI. And enterprises use Artemis with AI to be able to achieve these applications basically in a very fast manner. So we have thought through it, we have 12 years of experience. Some of the largest deployments of AI on the planet are on our platform. And 75% of our customers are in regulated industries. So we have had to deal with security, securing personal information, credit card information, financial information, healthcare information, and we have done it at scale. We have customers who put over a billion interactions. A single customer puts over a billion interactions on our platform. So we’ve taken all those learnings with all those scars on our back and distilled that into this new platform called Artemis.

Swapnil Bhartiya: When I look at your platform, your approach is governance first and that is actually happening all across. When it comes to those solutions, they try to tame agencies because the fact is that a lot of agents, they work autonomously, they’re not chatbots anymore, they’re actually taking decisions. And that’s when you need to have the harnesses, those guardrails, the governance is placed. Talk a bit of, when you say governance first approach, what does that actually mean in practice?

Raj Koneru: Governance is a very broad theme, right? Governance includes observability, traceability, guardrails, all of these things, right? While all of those are important, you obviously want to trace what happened. You want to see why the LLM made whatever decision it made. When it is being used as part of an application, you want to be able to get all the log files. All that is like, you know, basic foundation. The piece that is the most important, in my opinion, to not let these agents autonomously go and do the wrong things. In addition to guardrails is what we call eval. Evals are the way by which you test these agents to see if they are staying within bounds, are they achieving the objectives, basically, how accurate they are, basically. And can we improve those agents to be more efficient, to deliver better outcomes, basically. So when you think about the building process of an agent, and then after that the usage of that agent, using guardrails and evals give you the control that you want to have basically over that agent. Now, even when building an agent, there’s the deterministic part of the agent and there is the reasoning part of that agent. So in Artemis, we created a language called ABL agent blueprint language and we provide a capability called Arch. And you give Arch your goals, your SOPs and things like that, and it generates a multi agent application. But then based on your SOP, it learns what are deterministic execution business rules and what can be left to an LLM to interpret, make a decision or generate a response. So when it generates the multi agent application, it ensures that your business rules are tightly adhered to, that it’s not left to an LLM to make a decision. But when you need an empathetic response, basically at a point in time, based on the context, the LLM is able to provide that empathetic response. Or if it’s a straightforward reasoning kind of step, essentially you would then provide the context to the LLM with the information and the prompt which gets generated anyway, basically, and then the LLM makes the decision. So you’re basically both at design time, you’re able to restrict the LLMs and what they can do, but use them for what they do well. But then using evals, you test it, you test it with all kinds of use cases, I mean all kinds of test cases, basically to get you 100% coverage. And then you test it to see are my outcomes being met. Basically, if my outcomes are not being met, then I need to improve that. And that’s where another capability in Artemis called Auto Loop allows you to do that. Auto Loop is a capability where after the eval is run, it looks for failures in the agent and then regenerates the agent. Basically it is reflective learning and regeneration that then brings the agent as accurate and as efficient as it can be. And even after you go into production, Auto Loop will run on production data to see what’s not working, what’s working, and then regenerate the agent automatically, basically to make it more efficient. So the full loop is important. So governance is not just about administration and basically observability. It is about how you build these things, how you deploy these things, and how do you optimize these things. That’s the governance loop, if you will.

Swapnil Bhartiya: Thank you for explaining. And you’re absolutely right about governance can mean different things. Now if I ask you what are some of the governance mistakes that you have seen enterprise often make before they even realize that it’s a problem and they need a solution to actually fix it?

Raj Koneru: Well, I think it’s underestimating the evals, underestimating why they need to have evals, or having full coverage with the evals to be able to test every different scenario. So let’s take an example. Let’s say you’re a bank. You’re putting out an agent for customer service for the bank’s customers. The bank’s customer could come and ask about so many different things, right? First of all, the bank’s customer could be a teenager or could be a senior person, and somewhere in between. They could be from different backgrounds, they could be from different geographies. So the demographics are pretty vast essentially. And then the topics they may want to engage on, it could be about their bank account, their credit card account, their loan account, their mortgage account. They have a family event happening that they may need help from the bank. So the conversation contextually could shift in many, many, many different directions, basically. So when you build your agent, your multi agent system, you may build the functionality to cater to all of that with your business rules, with the tool calling integrations with your APIs and all of those things with MCP and all that. But then how do you anticipate all the different directions that the consumers may go into and topics that they may talk about, basically? And that’s where your evals come in, basically, because you give objectives as a bank saying, look, I need to be able to satisfy my customer, I need to answer their questions, I need to enable them to do the transactions that they want to do, basically. But at the same time, I don’t want these customers to have to fall out of that and have to talk to a human agent, which is the whole containment component, basically. Right. So there may be other goals that you may have. You know, when they’re talking about mortgages, I want to collect some information from the customer because I want to make an offer to the customer. That could be another goal. The evals help you evaluate all that, whether no matter which direction the customer may go into, it will be able to evaluate your agent to see, hey, is this giving us the full coverage in what scenarios is it failing? And then you’re able to focus on those scenarios and optimize the agent.

Swapnil Bhartiya: Is it also possible for you to just walk us through the lifecycle management on Artemis? What does it look like to manage an agent from the inception, from the moment it’s built and then when it needs to be shut down or retired?

Raj Koneru: Well, I mean the inception of any idea for an agent comes from a problem, right? A business problem. So what is the business problem? It could be that my costs are too high, it could be my sales are too slow or low. It could be that I’m not collecting money from my customers fast enough, whatever the problem is, right? And you sense an opportunity that with agentic AI you can make that much more efficient. Then comes creating of your, hey, how does your business run? Which is distilled into a standard operating procedure which includes your business rules, what systems that agent should connect to, all of that. Once that’s done, then much of everything else is automatic because Arch takes that SOP and automatically creates your multi agent system, the master orchestrator, each of the individual agents, what each individual agent does, it generates the ABL, which is the agent definition. It generates the tools, it generates the evals, it applies the guardrails basically and then runs the auto loop to make that agent extremely efficient. Basically then you go through your regular manual testing cycle if you have to, and then put it into production. So the speed basically is only constrained by you putting together the goals and the outcomes that you want and the systems that need to be used to achieve those goals and your business rules. So the upfront definition of what you want is the long pole in the tent. It’s not the build, the test, the deploy and the manage piece of it, which is very, very fast, that you can do that in a couple of days, basically. But then there’s the loop again, right? Once you put it into production, then you need to optimize it. So it’ll tell you right up front, hey, out of a million conversations, these 200,000 conversations didn’t go well. And it’ll tell you why, because it analyzes it, basically. And then it’ll make suggestions to you saying, okay, these are the changes that we need to make in the agents to make to handle those scenarios. And you review it and you say, yes, go ahead. And then it’ll apply those changes and then the next thing it’s back in production. Now, when do you retire an agent? I think you retire an agent when you don’t need it anymore, when the problem is solved, basically. I mean, you don’t see multi agent agentic systems getting retired yet, but I would imagine the only reason you would retire it is it’s of no value to me anymore, basically. So those could be small process agents, small employee workflow agents, not your big customer service agents or your big complex process agents.

Swapnil Bhartiya: When I was listening to you, you also talk a lot about multi agent coordination. How do you handle it differently when it comes to multi agent system that are built around single agent versus multiple agents?

Raj Koneru: Well, fundamentally it’s about breaking down the work, right? Basically at the end of the day, the user is interacting with the orchestrator. The orchestrator has cognizance of all the agents in the army, if you will, basically that are ready to do work. Why do you need multi agent? Because then you’re breaking down the work into logical pieces of work that each agent is able to do to bring efficiency. So the orchestrator sends it to agent one and then to agent two and agent three to achieve a goal or an outcome. Essentially this is the modern agentic architecture. This was even prevalent before AI, right? Whenever you had a program, a C program or JavaScript program, Python program, whatever, you always break it down into functions and the functions have functionality within them. With backend integrations, it’s similar in that regard, basically. But then imagine at an enterprise level, right, there’s a master orchestrator, there are sub orchestrators, there are multiple levels of agents. Now you’re replicating your employee workforce. So if you think about an org chart with the CEO at the top and everybody else in a pyramid structure, this is exactly the same. You’re creating a digital army, basically that works along with your human army to be able to achieve goals and outcomes, essentially. So it’s the delegation and the handoff. And the delegation and handoff could be to agents that are built outside our platform. So then there are protocols like A2A basically that you use to be able to communicate with agents built elsewhere. But essentially they’re all part of the same army, if you will, the org structure basically where some of the work that humans used to do has now shifted to these digital employees essentially. But then the digital employees, which are the agents, involve the humans at various points in time, basically during the workflow to take input, get approvals, review something before the workflow goes forward. So this is the new future that we’re going to deal with. Basically, you know, today or yesterday, employees like me or you were using these web based applications, mobile applications, to get data, put data in and execute a workflow. Now workflows are being executed by these autonomous agents which are adhering to business rules with the guardrails, with the evals and all of that. And we are either supervisors of those agents or participants in the workflow at a point in time. And that’s how I think you will see the future unfold. And for that you need a multi agent architecture as opposed to single agents that are not being orchestrated by anybody.

Swapnil Bhartiya: Of course almost everybody is on AI bandwagon these days, but given the challenges, of course the token cost, that’s a big discussion on tokenomics and of course governance, they have started to rethink and re-architect how they are doing AI. So a lot of companies are actually going back to the blackboard or for enterprises that are either still figuring it out or who are rethinking, what advice would you give them on how to govern and scale their own agents? With the governance first approach versus I mean we have solved a lot of problem, right? Security shift left, it is no longer someone else’s problem, it has become, you know, when you write the first line of code. So let’s talk about how they should look at governance because that is the most critical part, because building agents, running agents is the easy part, but actually taming them is the biggest challenge. And that could spiral to cost. It could spiral data leaks and so many other issues.

Raj Koneru: Well, I mean cost, cost is manageable. I don’t think it’s being efficiently managed in many situations, but can be efficiently managed. And in Artemis we automatically manage that. So when we build a multi agent system, each agent does a certain set of tasks and it could be like four or five tasks each agent does, right? So we automatically look in our model hub, in our model hub we have small LLMs, large LLMs, I mean small models, large models and in between. And we know their capabilities. So we choose automatically the right model for the right task to reduce latency, to still do things accurately at the lowest cost. And the tokenomics, the token cost is exposed through RMS and through Auto Loop. It automatically also optimizes the selection of models and model costs while they are high right now, basically over time, as the AI infrastructure grows in terms of scale, model costs are expected to commoditize and come down over a period of time, basically. So there is governance around that as well. Because you’re asking yourself the question, am I using the right model at the right cost for the right task? Which today people aren’t. People are like taking a simple task and giving it to ChatGPT or to Claude Opus, not needed. Basically you could even tap an open source model and get that task completed. So again, this is part of the learning exercise of enterprises over a period of time. And a platform like Artemis makes it easier for you to not worry about such things, basically because that optimization is already built into Artemis. So the governance comes from the quality of the platform that you’re using to do what you’re trying to do. It’s not just observability and traceability, it’s also efficiently being able to build and deploy these agents.

Swapnil Bhartiya: Raj, thank you so much for of course, breaking down the whole Artemis thing, governance from the ground up and to also, as you mentioned, cost and all those factors, which also means that companies like yours are the ones who are solving these problems. So once again, thank you so much. And those who are watching, please go and check out Kore AI to see how they can solve your problem. And once again Raj, thank you for joining me today and I look forward to have you back on the show.

Raj Koneru: Thank you, Swapnil, thank you.