AI may be everywhere, but most teams still have no idea what’s actually happening inside their large language models (LLMs). When latency spikes, costs balloon, or responses turn strange, engineers are left scrambling in the dark. The truth is, traditional monitoring wasn’t built for AI—and that’s why Orr Benjamin believes observability has to be redefined.

Benjamin, VP of Product at groundcover, has been building a new LLM observability platform powered by eBPF. His pitch is simple: if companies don’t get visibility into how their AI is behaving today, they’ll be blindsided tomorrow by outages, budget overruns, or even security leaks.

The eBPF Advantage
groundcover is using eBPF to see everything happening at the kernel level without requiring instrumentation. “Our sensor is at such a low level, it’s able to pick up all of that traffic,” Benjamin explained. That means teams can monitor requests to providers like OpenAI or Anthropic out of the box. Instead of adding more dashboards and more blind spots, engineers get real-time metrics and traces straight from the environment.

The benefits go beyond reliability. Costs, often the silent killer of LLM projects, can now be tracked with precision. “We’re able to tag each request with the token count on both the input and output, and translate that to cost,” Benjamin said. These metrics can then be aggregated by provider, model, or even by service within the organization, allowing teams to set budgets and alerts just as they would for CPU or memory usage.

From Cost Control to Design Decisions
Interestingly, teams aren’t just using this data for budget control—they’re making smarter design calls too. groundcover lets developers trace prompts, assess quality, and connect LLM results back to application-level issues like latency or rate limits. “You could go from the front end all the way through to the LLM requests and the backend requests,” Benjamin noted, making it possible to isolate where an application is breaking down.

That unified view is critical. Today, most large enterprises are already juggling four or five observability tools. Adding a siloed AI dashboard risks further fragmentation. groundcover’s strategy is to collapse these views, surfacing LLM metrics directly next to the infrastructure data developers already rely on. “You want less tools, not more tools,” Benjamin said. “Having everything in one place is the future.”

Security and the Road Ahead
Security is another blind spot LLM observability can help address. Because groundcover can analyze both prompts and responses, it can flag when sensitive data like PII is being sent to third-party providers. Crucially, Benjamin stressed, all customer data stays within their own cloud environment. “That risky data isn’t leaving for a third-party SaaS provider,” he said, underscoring the company’s bring-your-own-cloud model.

As for what comes next, Benjamin is cautious about predicting the pace of change in AI, but he sees LLM observability quickly becoming a standard. “We’re at very early stages, but the widespread use is going to be really interesting,” he said. His advice to teams just beginning their AI journey? Start monitoring from day one. Without observability, you’re effectively blind—and in a world where prompts, models, and costs change daily, that blindness can be fatal.

For enterprises betting on AI, the takeaway is clear: observability isn’t optional anymore. It’s the foundation for making AI reliable, cost-effective, and secure in production.