By It’s 2 a.m., and you wake up shivering, sweating, burning up with a fever. What do you do first?
You take a Tylenol. You drink water. You stabilize. You do not sequence the DNA of the bacteria in your throat first.
You treat the symptom to save the system.
In software, we do the exact opposite. A server spikes to 100 percent CPU – the digital equivalent of a 104-degree fever – and we do nothing. We watch. We wait for the crash, because we are obsessed with knowing the root cause before we administer the cure.
The thermometer is not the treatment. We built an entire industry that confuses the two.
Figure 1. The thermometer is not the treatment: the same 2 a.m. fever, with the order of operations reversed.
Everything we built, we built for a human reader
The dashboard, the pane, the alert, the retention policy: every one assumes that at the end of the pipeline there is a person who will look. That assumption produced two problems that look unrelated and are not.
The first is the bill. The model every vendor sells is ship-then-think: send everything you produce to a central cloud, just in case someone looks later. Pay on the way in, pay to store it, pay on the way out. Most of it is never read. That is the observability tax – paying for volume, not value.
The second is the needle in the haystack. We spent a decade buying observability, hoarding petabytes. Our dashboards look like the cockpit of a fighter jet. And when the system goes down? Chaos. Drowning in data, starving for answers. We optimized time to investigate; the only number the business feels is time to recover.
That is not incident management. That is analysis paralysis with a room named after it.
The market failure: a faster spectator
So the industry looked at the war room and the data pile and said: we have the answer. AI. Replace the engineer reading the logs with an agent that reads them faster.
Be careful. This is the faster-horse fallacy. Take a broken process – wait for the crash, then investigate – add AI, and you have not fixed it. You have automated the dysfunction. An AI that spends ten minutes writing a beautiful analysis while your site is down is a faster spectator, not a cure.
Figure 2. Adding AI to a broken loop: where the AI helps, and why time to recover does not move.
We do not need a faster pathologist. We need a doctor.
Three moves that change the loop
None of them is a better model.
- Stop collecting everything. Learn from the smartwatch. It does not track every cell in your body, and it does not mail your blood to a lab in another state. It tracks vital signs: latency, traffic, errors, saturation. Vitals stable, the system is healthy. Vitals drifting, act. The diet starts at collection, not at the invoice.
- Change the physics. If the smartwatch reads the vitals at the body, the doctor belongs at the bedside. Do not move the data to the AI. Move the AI to the data. An agent living next to the workload reads the signals locally, in milliseconds, with no egress: the data never leaves your perimeter. K8sGPT, a CNCF Sandbox project, already works this way – its analyzers run inside the cluster.
- Split the two jobs. Observability today bundles two different things into one expensive firehose: detecting problems in real time, and keeping raw data for forensics and audit. Detection at the source separates them. Once detection happens where the fault lives, the backend stops being your real-time nervous system and becomes what it always secretly was – long-term storage. And you do not pay hot-tier prices for that.
Figure 3. Move the AI to the data, and the backend becomes storage: one firehose doing two jobs, versus detection where the fault lives.
Kubernetes is ready for this. Observability flattens it.
Of everywhere you run software, Kubernetes is the best-prepared patient. The desired state is written down, a controller is already reconciling toward it, and faults arrive as typed objects with names – CrashLoopBackOff, OOMKilled, FailedScheduling – not a line you regex out of a log.
But acting on that takes what a dashboard never needed: the architecture. Which Deployment backs which Service. Which PVC that StatefulSet cannot start without. Which managed database sits outside the cluster and still takes the blame. That is topology, and it is the difference between restarting a pod and restarting the right one.
Observability treats Kubernetes as one more environment: hosts, containers, metrics and logs, normalized into the same shape as everything else on the invoice. That is the product working as designed, and it flattens away the structure autonomy runs on. A human at a dashboard supplies the topology from memory. An agent has none – it reads it from the cluster, live.
Which is why autonomy needs more sensors, not fewer. That sounds like a contradiction of the move before it, until you see which sensors: not more of the streams you already pay for, but the ones Kubernetes gives away – drift, cluster state and events, the audit trail, the live topology. Nobody centralized these, because no human was going to read them. They already sit next to your workload.
So the evidence carrying most of the answer is not the stream you are paying the most for. Metrics and logs stay, always on – they are how you feel the fever. They just stop being the whole case file. Right-size, not replace: nothing is deleted, the data is tiered differently.
What I am not claiming
The human keeps the last word for a good while yet. The useful shape is a gradient, not a switch: recommend only, approve first, autonomous – granted per fault class, as trust is earned. Every step reversible, every command in a transcript. And two boundaries: what senses should only read, and only one thing should write.
This also only works where the cure is known and verified. A novel failure still gets a human out of bed. Autonomy is earned fault class by fault class, not declared.
Why this gets urgent
We are entering an era where AI writes the code. Tomorrow’s systems will be too complex to debug in real time, with more change shipped faster than any war room can absorb.
We cannot keep building better thermometers. We cannot keep watching our systems die in high definition.
So, a question for anyone adding AI to their incident process this year: did it change what happens before the investigation, or only how fast the investigation runs? One of those is a cure. The other is a faster spectator.
Nati Shalom has spent 25 years building distributed systems and SRE platforms. He is co-founder and CTO of DataAgent.
KubeCon + CloudNativeCon is taking place in Salt Lake City, Utah from November 9 to 12. Register Now!
