Author: Ranjan Parthasarathy, CEO, Devtron
Bio: Ranjan is a seasoned entrepreneur who founded ClusterUP, LogigIQ, which was further acquired by Apica, and now serves as CEO of Devtron Inc., where he is on a mission to democratize Kubernetes adoption with Devtron Platform.

And yet, the way we observe and operate Kubernetes often reflects an outdated mindset: applications in one silo, infrastructure in another, and stateful workloads treated as afterthoughts. The result is a sprawl of dashboards, the official Kubernetes dashboard, Lens, Headlamp, Rancher, MLflow, Kubeflow, and many others; each offering a slice of visibility, but rarely the whole picture. For AI/ML teams, this fragmentation is especially painful: model training metrics live in one system, cluster resources in another, model versioning in a third, and cost attribution nowhere at all. The proliferation raises an important question: in an operating system designed to unify, why do we continue to treat application, infrastructure, and data state as separate entities?

Too often, organizations still treat these as separate domains: applications belong to developers, infrastructure to platform engineers, and stateful systems to database administrators. Yet in Kubernetes, they are the same fabric. StatefulSets, PersistentVolumeClaims, GPU node pools, model serving endpoints, and ingress rules aren’t “just infra”; they’re the very foundation on which modern AI/ML applications and data-intensive workloads live and die. When you split the three, you invite chaos.

The Illusion of Separation in the Age of State and AI

Traditional IT operations drew a hard line between applications and infrastructure. Apps lived closer to the business; infrastructure was “just plumbing”; and databases were isolated kingdoms. Kubernetes blurs all these boundaries. A StatefulSet pod crash is an application issue, but it may stem from PersistentVolume performance bottlenecks or node affinity misconfigurations. A GPU allocation failure isn’t just an infrastructure concern, it directly halts a critical model training job that impacts business outcomes. An out-of-memory error during inference may trace back to model size, batch configuration, or inadequate resource limits.

Despite this, many teams still toggle between an “infra dashboard,” an “app dashboard,” a “storage dashboard,” and separate MLOps tooling. For AI/ML workloads specifically, the disconnect is severe:

• Data scientists track experiments in MLflow or Weights & Biases
• Platform engineers monitor cluster health in Prometheus/Grafana
• Storage teams manage persistent volumes separately
• FinOps teams struggle to attribute GPU costs to specific models or teams
• DevOps manages CI/CD pipelines without visibility into training job dependencies

This fractured experience doesn’t just waste time; it obscures the fundamental truth that in Kubernetes, application logic, infrastructure resources, persistent state, and AI/ML pipelines are inseparable aspects of the same system.

Operational Chaos in the Stateful & AI/ML World

The explosion of Kubernetes and MLOps dashboards has created a paradox: the tools built to reduce complexity often amplify it.

• One tool helps visualize StatefulSet ordering and PVC binding
• Another manages GitOps deployments for microservices
• Another focuses on GPU utilization and node pool health
• Yet another track model training experiments and hyperparameters
• Another layer in model serving and inference monitoring
• Another handles data pipeline orchestration

Each serves a purpose, but together they fragment the operator’s view. Teams jump between windows, reconcile conflicting metrics about storage performance, GPU allocation, and model accuracy, and stitch together a mental model of what’s really happening. For AI/ML workloads, this operational chaos is especially costly:

• Training jobs fail silently due to PVC mount issues discovered hours later
• GPU resources sit idle while jobs queue because of namespace quota misconfigurations
• Model drift goes undetected because inference metrics aren’t connected to cluster performance data
• Data pipeline failures cascade through training jobs without clear visibility into the dependency chain

This slows down incident resolution, complicates governance, delays model deployment cycles, and adds cognitive overhead at precisely the time platform teams are expected to accelerate AI/ML innovation.

The Cost of Operating in Silos: The AI/ML Tax

Operational visibility alone isn’t enough anymore. As Kubernetes becomes the operating system for AI/ML workloads, the financial stakes are astronomical. Without embedded FinOps practices tailored to stateful and AI/ML workloads, organizations risk financial disaster.

Consider the following scenarios:

• Data science teams spinning up multiple training experiments with A100 GPUs at $3+ per hour, with no cost accountability or automatic shutdown
• Model serving endpoints over-provisioned with expensive GPU instances running 24/7 for models that serve sporadic traffic
• Persistent volumes proliferating across namespaces—snapshot chains, orphaned PVCs from deleted StatefulSets—accumulating massive storage costs
• Shared GPU clusters where one team’s hyperparameter sweep monopolizes resources and generates six-figure monthly bills
• Data preprocessing pipelines running on premium storage tiers when object storage would suffice
• Model registries storing hundreds of experiment checkpoints with no retention policies

FinOps for AI/ML isn’t just about after-the-fact reporting; it’s about building cost awareness into every layer:

• Per-experiment cost tracking: Attributing GPU hours, storage I/O, and egress to specific training runs
• Model TCO visibility: Understanding the full cost of a model from training through serving, including storage, compute, and inference requests
• Rightsize recommendations: Automatically detecting over-provisioned inference endpoints or underutilized GPU node pools
• PVC lifecycle management: Identifying orphaned volumes, expensive storage classes, and snapshot bloat

In the AI/ML world on Kubernetes, cost is neither purely an “application” concern, an “infrastructure” concern, nor a “data” concern. It’s all three, inseparable, just like the training jobs, the persistent state they depend on, and the GPU clusters that power them.

The Stateful Challenge: When Pods Aren’t Cattle

The rise of stateful applications on Kubernetes introduces unique operational challenges that compound the visibility problem:

• Identity and ordering matter: StatefulSets require careful pod management, persistent network identities, and ordered deployment/scaling
• Storage performance is critical: Database workloads demand low-latency, high-IOPS storage that’s vastly different from ephemeral container needs
• Data durability and backup: Losing state means losing business value, but backup strategies across distributed StatefulSets are complex
• Affinity and anti-affinity rules: Stateful workloads often require careful pod placement for performance and resilience

For AI/ML specifically, these challenges multiply:

• Training checkpoints: Models training for days or weeks need reliable checkpoint storage and recovery mechanisms
• Dataset caching: Multi-terabyte datasets must be efficiently cached close to GPU nodes without exploding storage costs
• Model artifacts: Trained models, feature stores, and embeddings need versioned, high-performance storage accessible to inference services

Traditional application-focused dashboards weren’t built for this complexity. They show pod status but not PVC attachment states, resource utilization but not storage I/O patterns, or deployment health but not StatefulSet rolling update progress.

From Visibility to Agency: AI/ML Automation

Even a unified dashboard is only the first step. The future of Kubernetes operations for AI/ML workloads lies in agentic workflows, platforms that don’t just show you what’s wrong, but understand context and take intelligent action:

• Detecting a failed training job due to OOM errors and automatically adjusting memory limits based on historical patterns
• Identifying idle GPU resources during non-business hours and scaling down inference replicas or pausing development clusters
• Auto-recovering StatefulSet pods stuck in pending state due to PVC binding issues
• Proposing model compression when inference costs exceed thresholds relative to accuracy gains
• Orchestrating data pipeline retries when upstream storage systems experience transient failures
• Optimizing experiment scheduling by batching jobs to maximize GPU utilization and minimize queue times
• Triggering model retraining when drift detection agents identify degraded performance

This shift turns dashboards from static control panels into intelligent copilots that understand the full AI/ML lifecycle, reducing toil and enabling platform teams to focus on innovation rather than firefighting.

A Philosophy Shift for Platform Engineering in the AI Era

The Kubernetes ecosystem doesn’t need more dashboards or specialized MLOps tools. It needs a new philosophy: treat application logic, infrastructure resources, persistent state, and AI/ML pipelines as one unified system, because Kubernetes makes them one.

That means:

• Unified visibility that spans from model training metrics to GPU utilization to PVC performance to inference latency
• Embedded FinOps that tracks the true cost of every model from experiment to production
• StatefulSet-aware operations that understand pod identity, storage dependencies, and ordered scaling
• Agentic workflows purpose-built for AI/ML lifecycles, training, validation, deployment, monitoring, and retraining
• Storage-conscious architecture that optimizes for the massive I/O demands of modern ML workloads

It means reducing the number of windows operators need to keep open and embracing Kubernetes as the operating system for the entire AI/ML stack, not as fragmented domains controlled by different teams.

For platform engineers supporting AI/ML workloads, this is more than a tooling question, it’s a fundamental mindset shift. Success won’t come from juggling a dozen specialized dashboards for experiments, infrastructure, storage, and costs. It will come from embracing fewer, smarter, and more holistic platforms that understand the full lifecycle of stateful, data-intensive, GPU-accelerated AI/ML workloads.

Building the Unified Platform for AI/ML on Kubernetes

Devtron is one such platform that brings applications, infrastructure, and stateful workloads into one unified system, with specific capabilities for AI/ML teams:

• Developer productivity: Streamlined workflows from code to production, including model training and serving pipelines
• Operational efficiency: Reduced toil through automation that understands StatefulSets, PVCs, GPU scheduling, and ML-specific deployment patterns
• Cost visibility: Granular tracking of expenses by cluster, namespace, workload, and critically, by model experiment, training job, and inference endpoint
• Agentic workflows: Deep integration that understands CI/CD pipelines, model registries, deployment dependencies, and can troubleshoot issues across the full stack faster than ever

For organizations running stateful applications and AI/ML workloads at scale, the platform provides the unified observability and intelligent automation needed to manage complexity without adding more tools to the sprawl.

If you’re at KubeCon, stop by Booth 1641, we’d love to continue this conversation about the future of Kubernetes operations for stateful and AI/ML workloads, and show you how Devtron can help you manage the full complexity of modern cloud-native applications.

KubeCon + CloudNativeCon North America 2025 is taking place in Atlanta, Georgia, from November 10 to 13. Register now.