Observability Without Feedback Fails AI Code | Bob Quillin, ControlTheory | TFiR

Swapnil Bhartiya: I don’t even have to say this. As you all know, that more and more code running in production today is written by AI. And ironically, the teams who ship and run that code don’t fully trust it yet. In a recent survey, over 200 senior SREs and DevOps leaders said they lack confidence that AI generated code behaves as expected in production. Because what happens in production rarely feeds back to the agent or engineer who shipped it. There is no accurate feedback loop. Control Theory calls that missing piece controllability. And they have built Digital8 to close that loop. And today we have with us once again, Bob Cullen, CEO and co founder of Control Theory. Bob, it’s great to have you back on the show.

Bob Quillin: Yeah, great to see you swap, Neil. And great to talk to you again.

Swapnil Bhartiya: It’s my pleasure. Of course, I’m going to talk about controllability. I’m going to talk about this slate. Before that, of course, when you created the company, we had known each other for a very long time. We talked about, I think about last year or so. Talk a bit about how the company has evolved and what brought you to Dstl8.

Bob Quillin: So when we founded Control Theory, really the idea was how do we rethink observability? And the core idea was observability really needs a feedback loop. And that’s how we came up with the idea of Control Theory and Control Theory. The science has two elements. One is observability, but one is also controllability. And that creates that control system, that creates balance in the world. The best example probably is a thermostat system that we all have. We all trust that to work and we don’t. We’re not always asking it to change it up, change it down. We set it. Our surface level is the temperature and the thermostat adapts as temperature goes up and down. And that’s a. That’s really the most simple but most understandable control system that’s out there. So we thought, you know, really, we need to bring that back into. Bring that into observability and balance out this idea of controlling with a feedback loop back to the people, the systems that can actually fix the problem. When we started the company, it was a couple years ago, that was a core idea, really addressing that core belief that we needed to bring that feedback into observability itself. And then with the introduction of AI coding, cloud code, the high velocity and volume that began to head downstream into all the DevOps and SRE and operational tools that are out there, it became clearer that not only was this a good idea for just observability, but it was basically a requirement to really help drive the AI software development lifecycle in a much more effective way. You know, we kind of say, you know, observability without feedback is just watching. So you really need to have that feedback loop, feedback into the system and do it in a more effective way. If you think about the software development lifestyle collapsing going much quicker, we’re getting much more effective at how we deliver code at higher volumes. But downstream, not a whole lot’s changed. We’re still collecting all that information, putting it in big data lakes and trying to build dashboards on top of that. So while a lot of stuff has changed upstream with the coding and the approach to AI development, AI engineering downstream we need to work at that AI speed. So that’s why we call Dstl8 the runtime feedback loop for AI speed engineering. So that’s the idea and I think actually the stimulus or the, what’s going to change observability once and for all. Here is this introduction of the AI era, the agentic era, really forcing us to rethink how we do it so it fits better into how we want to develop code and product in the future.

Swapnil Bhartiya: Now let’s talk about survey. It showed that SREs and DevOps leaders don’t fully trust AI generated code. The fault in my opinion is not with the AI itself. It’s like hiring a junior developer or a chef and then expect them to create production grade tools. No, you, you do need human oversight. So can you talk about the trust wall? What is behind that trust wall? The code, the LLM, the harness, governance. Why is there this lack of confidence in something that your own team helped create?

Bob Quillin: Yeah, because I think runtime production is the ultimate arbiter of whether you’re successful as a deploy, how it’s being used by customers, how it’s running in the infrastructure. You run it on behavior changes. There’s a lot of non deterministic behavior in AI code that you didn’t expect. So we kind of look at trust, there’s a whole trust chain and the idea that a lot of people look at their telemetry and are beginning to, you know, look at the data as a source of the truth. But there’s a telemetry trust issue out there and I think that’s a huge, you know, if you can’t trust your telemetry, how can you trust everything that’s downstream from it determining what’s working you know what’s hidden in my filtering or sampling out information I don’t need. Am I paying too much? Am I paying a telemetry tax? And storing all this data and with the, and not being able to use it effectively. And so I think a lot of the trust begins to look downstream, especially as we’re running this, dealing with issues in production. And we think it begins with trusting your telemetry and feeding that the right signal, reducing the noise, getting that signal down. That’s why we talk about distillation being critical for this and distilling out the information that you need to drive a model. We have our own custom fine tune model we call Mobius. But you’ve got to have the right context. Everyone knows about context Windows. If you put a fire hose of logs or metrics or traces into an LLM, it’s going to blow up. So that model doesn’t work anymore. People have tried it cost a lot of money, it’s not effective. So you got to feed the right context, right prompts. We all know about the right prompts as it comes to making your agents work for you. So having the right telemetry distilled down and controlled and built for the environment you’re going into is kind of rethinking once again. Flipping observability on his head. Before the last decade, when it was human generated code and humans reading dashboards and humans writing monitors in a very deterministic way and writing alarms, everything just got more known. Now we’re dealing with a much more different timeframe. So the idea that now agents are helping write the code, somebody’s writing all the code, you’re managing that process, how do I get that information back to the agent or engineer that’s writing it? So the idea that telemetry distillation driving the right model, getting that information back, helps to turn observability on its head. So I don’t have to distort everything. I don’t have that telemetry tax. We talked about telemetry trust, but it’s also the tax of paying for all that telemetry ingest, telemetry indexing, telemetry retention. People hate the cost of observability. That’s like one of the other things that got the company started was like we think architecturally we could make observability better by adding controllability and feedback loops. But also no one likes the pricing model. You’re paying for storing all this data you don’t need. You’re writing dashboards, you’re doing all the work and all the alarms on top of that. And this is a world we’re moving into where I don’t need all that data. I need the data that’s going to drive the model, make the right decisions and do it quickly, proactively and not reactively. So all those things kind of are how distilled like us started where we’re actually right now as a product distilling telemetry at the source, moving it into a model that can actually solve problems without you having to write those alarms, without the human necessary having to be the bottleneck in the observability process and then getting it directly back into the agent in the cloud code or whatever you’re using to actually fix it, whether it be an infrastructure or code problem, whatever’s happening in runtime. So there’s a lot of moving parts, but it comes down to trusting your telemetry, distilling and getting that signal out and really beginning to rethink why do we collect all this data when really what the information, what we need is really 1 or 2% of what’s being generated.

Swapnil Bhartiya: Now when you look at controllability, it looks critical right now. What cap did you see that led you to start the company and that controllability became the core focus?

Bob Quillin: We looked at a whole range of issues as we kind of looked at the market. We’ve been in observability our whole careers and monitoring and application performance management. And we saw where the AI SRE market was developing and saw all these different sort of movements happening and some movement not happening. But it was still based upon the premise that I’m going to collect all the information, I’m going to put in a big data lake, I’m going to bolt on a, maybe an agent on top of that. That’s kind of the new model, these bolt on agents that go on top of your existing monitoring. So I’m still paying for the old monitoring now I’m paying for the agent itself and I’m paying for all the symmetry. I don’t need where I really want to pay and look at the incidents and the outcomes. I want to, you know, that’s really where a lot of the AI tools are looking right now is like I want to pay for the outcome. I want to, like here’s the incident that I’ve solved. I want to be able to track that. That’s the outcome I’m looking for. I don’t want to pay for the telemetry. I’m ingesting, I want to pay for the results. So we saw cost issues, we saw architecture issues, we saw a lot of dissatisfaction with the way the system was built. And fundamentally we believe that all this is important from a customer perspective. But the AI software development life cycle is compressing. It’s moving much more quickly and observability needs to be a core part of that. And then controllability with the feedback loop is the answer. Just feedback into the agent that can fix it. And that software development lifestyle can move much more quickly. Otherwise observability becomes a boat anchor dragging down the whole speed and velocity and volume you’re trying to increase. So you get a higher quality, higher speed just by looking and rethinking of how observability has written in the past. So that that’s really where Dstl8 came from. We distill information at the edge, we correlate it, we help you remediate it, and then you remember. We have a knowledge graph that remembers that. So you don’t have all that tribal knowledge out there too. It remembers and builds a memory around what the problem is. And it’s all driven by one of the core concepts which I kind of started with, which is telemetry distillation. And we can go to that in a little bit more detail, but I’ll save that for next question maybe so they can take a deep breath.

Swapnil Bhartiya: You have called telemetry distillation the core technology behind Dstl8. I love that name too. So many connotations. Where did that come from? And why does it matter so much to you to or to control theory and to the whole ecosystem?

Bob Quillin: Yeah, the telemetry distillation is really our core invention that drives everything upwards. It drives a better model. Our Mobius AI model we built and it drives is continuous. So one of the things that we love about we wanted to build was a continuous distillation and continuous way to look at information without you having to actually write alarms and understand what’s going to happen in advance. One thing we found with almost every customer we work with is that we find a lot of hidden problems and unknown issues. And these issues are not just LLM issues, but these are also problems that have been out there for a while that no one’s actually seen because we’re watching that runtime, we’re actually looking at all the behavior. So telemetry distillation is not filtering and it’s not sampling. It’s actually looking at every log line. It’s looking at the contents of that log line and trying to determine a new signal that we talk about, which is called sentiment and sentiments, used in a lot of other business processes around PR and brand awareness and how you’re looking in the news and maybe customer success models, et cetera. How is the world perceiving us? But logs are language and we can actually distill out issues that are hidden inside those logs, because any engineer or any software developer knows that they write info logs, debug logs as different severity levels, but embedded in those are a lot of key sets of information that are negative sentiment. So we actually can pull out that information and know that this log line has a problem in it. It may not look like it as an information log level log or debug log, but we can pull that out and that’s a sentiment measurement. We also can look at anomalies and changes that happen. We’re doing this locally, at the edge, in the customer’s environments. We’re not sending up all the logs, we’re sending up only the sentiment, the anomalies and the logs that are related to that particular issue. So we’re really distilling out 98, 99% of the problem and sending out the signal from the noise. So that needle in a haystack, we can pull that up. So we do this continuously and that’s really called continuous telemetry distillation. And telemetry distillation is the start of this. It helps to drive a better model and a better analysis. So it reduces the context and drives the solution into an area where you can actually go remediate it. We can pull in a code graph, we can pull in topology, relate that to the incidents we’re seeing and the distillation of telemetry. And all those then relate to better problem solving, better incident detection, and pinpointing where the problem is, helping the end user or the agent go fix it. And either we can pull that into cloud code and they can remediate it automatically if they like, or they can get cloud code working out immediately and you could confirm and validate that information. So the idea is closing that loop, getting that runtime information, pulling it out continuously, getting that information back to an agent or an engineer that can fix it. And that bottleneck is still out there. It’s been out there for 20 or 30 years. Every team we talk to talks about, I wish our development team could get better information from the ops and production side. Because every time there’s a problem, we go back and forth, we go through a post mortem. They say there’s not a problem, we say there is a problem, they say there is a problem, we say there’s nothing. And there’s no way to get accurate, timely information to help close that loop and that development cycle. So that’s where we go back to the software development lifecycle. In trying to improve that.

Swapnil Bhartiya: AI is not part of the software development lifecycle that carries out of the bag. It’s not going to go back there. So we have to accept that and move forward. What challenges does that create for traditional monitoring and observability tools which were built for a word where humans wrote and read every line of code?

Bob Quillin: Yeah, it’s interesting. Look back five, 10 years, everything was in balance in terms of the speed of which product was being written, was governed by the speed that engineers could actually develop the code, deliver the code, push it out to production. And we automated a lot of those processes as much as we could. But the bottleneck was still on the human side. In comes AI. It creates higher velocity and higher volume. And there’s a lot of teams that have begun to really embrace that. Most of them have. We’re seeing tremendous increase in the amount of code coming downstream, product coming downstream. So the efficiencies is way up there. But what that challenge is then is everything downstream was built for a different generation. It was built for that, the pace of human generated code. It was built for dashboards that were being monitored by humans. It was built for alarms and monitors that I find a problem, I write alarm for that. So next time I get alerted to it. So everything came back to the human itself, which is appropriate. We built a lot of tools and I did myself, and I think we moved the market forward. But now we’re in a brave new world and rethinking of how we move the concept of velocity and volume and adapt to that in a different way. And that’s where we come back to really the core concept of feedback loops and controllability and getting that into the life cycle of the agent itself and the AI software side. So it’s not dev and ops anymore, where an ops team is delivering information back and is delaying responses. You could actually have a very tight feedback loop now by continuously distilling your information, looking at problems, feeding that back in, and you get fairly instant responses of how you can fix problems before they even become problems. So you’re much more proactive, you control the cost. So you basically checked off two issues that were already exacerbated by AI downstream. You look at existing observability tools. So we think it’s a brave new world in terms of where monitoring observability is going to go. And we think, you know, we may not talk about observability in the future. It should be all about controllability and software development loops and speeding those up and making those work more effectively. So I think it’s embracing those new worlds and changing, you know, not just how everything’s been architected and reimagining that for a new world where the velocity and volume of AI are going to overwhelm everything downstream. And you can’t just bolt on these old solutions. We’ve already changed how we’re developing code. Let’s continue that process all the way through and look at how we can improve and optimize what’s going forward and make observability and controllability part of that cycle.

Swapnil Bhartiya: Now, if we just keep AI aside for a bit. Can you talk about, are teams actually happy with their current monitoring and observability tools or they were using whatever was available and they were content with that. Because before AI, most tools only tell you what’s going on, which I feel lack of actionability. I don’t think that’s enough.

Bob Quillin: Right. Yeah. Like we said, observability without feedback is just watching. And I think every observability vendor recognizes the challenge that’s coming forward. And even the most traditional legacy vendors are making changes, but architecturally they still rely on collecting all your information, putting in a big data lake, charging you for that, building dashboards on top of that, and building it for humans. And then actually you can buy an agent that goes on top of that too. So what people are dissatisfied with and have been for a while is the cost model, the telemetry tax I talk about. And then that goes back to telemetry ingestion, telemetry indexing, telemetry retention, all those things. So to solve the problem, you got to start at the source and distill that telemetry, which is what we said, let’s go back to the source of the problem. That changes the cost model. It changes how the model can then fit into the AI world. And it works on a continuous basis. It doesn’t require a reactive model. It can be proactive because it’s working continuously without a human having to engineer and model everything. They can guide it, they can try to figure out, push it, put some surface levels and boundaries and guardrails around it as they should. But the idea is that AI should be working for you. It shouldn’t be something that is an afterthought, comes in later and looks at what’s sitting in your database and tries to figure out what happened a day or two later. So yeah, we think that rethinking from the bottom up and totally rebuilding this concept is what’s required. It’s a new opportunity. It’s contrarian. We admit that it’s not the way people think about it. And we still run into a lot of customers and engineering teams that are hoping that the old stuff they have is going to work with the new AI coding they have. And now they’re starting to see some cracks in that. But then there’s new teams that are coming in and new people looking at hey, AI is going to be across the board. We got to look at how that’s affecting the entire organization, not just development side. So operations is under pressure too. Platform teams, DevOps teams, ops teams are, everyone’s facing constraints, hiring wise, headcount wise. And people are looking for more efficiencies too. So we think that all those factors play into an opportunity to really drive observability to the next generation. Adding controllability, adding that feedback loop and really focusing on what’s next. So that’s where we come in.

Swapnil Bhartiya: Now let’s just get into the weeds a bit. How does Dstl8 look at logs across the whole system and let’s not look at it mechanically and clinically.

Bob Quillin: I’ll go back to telemetry distillation. We’re pulling out a very unique signal right now called sentiment. We talked about sentiment and it really lets us identify problems that are both obvious and you want to catch all the ones that you should catch, but also the ones that are not so obvious, they call them silent or hidden problems. A good example is LLMs. Their APIs return something called HTTP 200. OK, so you get a lot of these okay responses, but within those logs there’s a series of invalid prompts, issues around payload refusals, there’s a series of errors that happen inside a response that’s green. So the response is green, but within there there were problems. So basically the LLM is working the way it’s supposed to work, but it’s failing to give the customer the response it’s supposed to give. It could even be an invalid prompt error or someone wrote something in wrong. So the LLM responds appropriately, but still the customer’s not happy. But the API comes back as okay. So within there, we could actually see that there were problems in what is typically an okay response, a green response. This is fine. It responded successfully in the old world. Thumbs up in our world. We look at that, we distill that down, we say, hey, there’s an error hidden inside this. And you flag that and you could fix that for the customer because there’s still an issue. Even though the LLM might have operated the way it’s supposed to, the customer saw something different. So there’s hundreds of thousands of these different kinds of issues that percolate through not just LLMs, but hidden code issues, hidden behavior issues. We think that sentiment is core to looking at what’s happening and being able to tell you there’s problems I didn’t expect to see. And we can then grade those from minus one to one. So we give you a heat map of those. We feed that sentiment up to our agent Mobius, who then can use that to find problems. It combines that with maybe an anomaly that it detected, something changed, maybe it’s something in the topology that we saw. We pull in a code graph and figure out where the code and that log actually line up. So all that stuff could be correlated inside the model itself to say, hey, here’s the problem, here’s where we think it is, and this is how you should solve it. And straight into remediation loop from that side on. So the ability to distill and correlate that at the model, remediate that at the agent, and then keep that history. That’s the cycle that we look at, distill, correlate, remediate and remember. So that’s really driven from the beginning through the telemetry distillation, a better model of reasoning through our agents and then actually getting that back into the agent or engineer that can solve the problem as quickly and continuously as possible. So the loop drives and fits into the cycle that’s out there today. The software development lifecycle is moving quickly. We don’t want to slow it down. We want to speed it up if we can, or at least work at the same pace.

Swapnil Bhartiya: Dstl8 is now generally available. What can teams do with it on day one? And how do they get started? How much hand holding is needed from ControlTheory? Or they can just get it started on their own?

Bob Quillin: Yeah, it’s built to plug in across a variety of different sources. It’s great with Kubernetes and OpenTelemetry, it was built on OpenTelemetry core. So that’s really where a lot of the company and where we came from. Cloud native CNCF. We’ve been going to Kubecon for a while. So two things we know really, really well are Kubernetes and OpenTelemetry. We have a great CloudWatch collector that can push information into what we do. So it’s very seamless. We work with Google, we also work with some of the newer platforms like Supabase and Vercel and Cloudflare. So we want to have that full runtime plane and collect all that information as much as possible and correlate across those multiple sources. So we can do that. But each of those are very simple. It installs very quickly. We have a command line interface, a CLI if you want, prefer to use that too. And then you could hook it into our MCP server and skills that work within your agent itself, but seamlessly can be up and running within five minutes and it actually within the first half hour, hour, usually finding a problem or two that will surprise teams and they’re like, huh? And they call someone over and like, what’s happening there? There’s a type error, there’s an API call that’s not working. There’s something happening in the database and they see problems that they recognize that they didn’t recognize before. And people are using it also in staging, which is interesting. So we’re not cost prohibitive like many of the other tools to be able to deploy us both in a staging environment where they have runtime testing and also when they move it out to production. So we’re also a nice safe solution within staging and very cost effective there too. So many people are testing a lot of the AI environments within their staging first before they move into production. So that creates a nice little one, two punch. So it’s a very simple to use across many different kinds of sources. It could work for a team. The approach is that you can set it up for multiple teams. There’s workspaces. So each team could have a different workspace or maybe it’s your front end, your back end, your staging, your productions. There’s different workspaces that have different views too. So it’s set up to be partitioned how you want to have the world looked at from your perspective and from your infrastructure and your applications perspective too.

Swapnil Bhartiya: So Bob, thank you so much for breaking this down and everybody who’s watching it, please go and check Dstl8 control theory.com and Bob, thank you once again and I would love to have you back on the show.

Bob Quillin: Great. Thank you. Yeah, we’ll see you all at Kubecon.