By Every GPU cluster running AI training or batch queue workloads carries a quiet tax: watcher pods that run continuously, consuming resources and budget even when there is zero demand. Until now, Kubernetes had no native way to scale a deployment all the way down to zero pods while still reacting automatically to incoming work. Teams either over-provisioned or built fragile external scaffolding to approximate the behavior.
In this interview on TFiR, Dipesh Rawat, Kubernetes v1.37 Release Lead at CNCF, walks through three features in the v1.37 release that directly address cost, cluster resilience, and continuity for long-running AI jobs.
Guest: Dipesh Rawat, Kubernetes v1.37 Release Lead at CNCF
Show: TFiR
Here is what every platform engineer and AI infrastructure team needs to know.
Technical Deep Dive
Q: What does HPA scale to zero in Kubernetes v1.37 actually eliminate, and what is the real cost impact for GPU workloads?
Dipesh Rawat, Kubernetes v1.37 Release Lead at CNCF, explains that without this feature, teams always needed at least one pod running continuously to watch for event-driven triggers or external metrics, even when there was no active demand. With HPA scale to zero graduating to beta and enabled by default in v1.37, that always-on watcher pod is no longer required: workloads can hook into external metrics such as queue depth and spin up only when demand arrives. Rawat notes that in an environment where GPU and memory availability is constrained, eliminating idle resource consumption has a meaningful cost impact, particularly for AI workloads running on shared clusters.
“This feature is going to save a lot of clusters and GPUs, especially in times where there is a shortage of RAM and memory and there are a lot of challenges in getting hold of those.”
Dipesh Rawat, Kubernetes v1.37 Release Lead, CNCF
Q: How does HPA scale to zero work with event-driven or queue-depth metrics in Kubernetes?
Rather than requiring a dedicated watcher pod to poll external systems, HPA scale to zero allows the Horizontal Pod Autoscaler to consume external metrics, such as queue depth or event-driven signals, natively. When the metric crosses a threshold, Kubernetes brings the workload up from zero replicas. When demand subsides, it scales back to zero. Rawat describes this as an out-of-the-box capability that replaces bespoke scaling scaffolding that teams previously had to build and maintain themselves.
“You can hook up to external metrics such as some event-driven or some queue depth, and based on the demand, as soon as the demand comes up the workload automatically sets up.”
Dipesh Rawat, Kubernetes v1.37 Release Lead, CNCF
Q: What failure mode does Resilient Watch Cache fix during Kubernetes API server startup and recovery?
When a Kubernetes API server restarts, it must rebuild its in-memory watch cache before it can serve requests reliably. On large production clusters, this rebuild takes time, and clients connecting during that window previously received timeouts rather than actionable responses. Resilient Watch Cache, now stable in v1.37, gives the API server a mechanism to reject those premature requests with a proper HTTP response, allowing clients to retry gracefully while the cache finishes initializing. Rawat emphasizes that this closes a well-known failure mode that made maintenance windows and unplanned restarts more disruptive on large clusters.
“For a larger cluster this stable feature is going to enable them to have a more resilient way of handling the outage or the maintenance instead of taking a hit and making that outage window longer.”
Dipesh Rawat, Kubernetes v1.37 Release Lead, CNCF
Q: Why does Resilient Watch Cache matter more as Kubernetes clusters grow larger?
The larger the cluster, the longer the API server cache rebuild takes after a restart, and the larger the window during which clients experience timeouts. Rawat notes that most production clusters are large, which makes this a broadly applicable stability improvement. The stable graduation means the feature ships enabled by default and teams do not need to opt in or configure anything to get the protection.
“Most of the production clusters are large, and clients are going to connect to those and they are going to get timeout. Now with resilient watch cache there is a way for the API server to reject those requests and give an appropriate HTTP response so that the clients can retry.”
Dipesh Rawat, Kubernetes v1.37 Release Lead, CNCF
Q: What does pod checkpoint and restore in Kubernetes v1.37 alpha unlock for interrupted AI training jobs?
Pod checkpoint and restore, landing in alpha in v1.37, allows a long-running pod, such as an AI training job, to save its in-progress state at the point of disruption. Instead of restarting from the beginning when a pod dies due to a node failure or cluster disruption, the job can resume from the checkpoint, preserving hours of completed computation. Rawat frames this as particularly valuable for AI and ML workloads where training runs can last hours or days, and a forced restart from zero represents a significant loss of compute time and cost.
“If your job has run for two hours and it died at that two-hour point, you have to again go back to point one and start from zero. What this feature gives you is a checkpoint, so at the point where the job goes down you can restart the job technically from there, keeping all your progress.”
Dipesh Rawat, Kubernetes v1.37 Release Lead, CNCF
Q: How close is pod checkpoint and restore to production readiness, and what should teams do now?
Rawat is clear that the alpha designation means the feature has not yet passed the graduation criteria that the Kubernetes project requires before calling something production-ready. The timeline to beta and stable depends on community feedback and whether the feature meets the defined acceptance criteria. He encourages teams to enable it, test it against their use cases, and submit feedback so that real-world results can accelerate the graduation process.
“We highly recommend that you turn on this feature, try to use this feature, and please give us feedback that if it can be improved further to improve your use case.”
Dipesh Rawat, Kubernetes v1.37 Release Lead, CNCF
Resources & Documentation
- CNCF, Cloud Native Computing Foundation, home of the Kubernetes project and release process
- Kubernetes Horizontal Pod Autoscaler, official documentation including scale-to-zero configuration
- Kubernetes Blog, release announcements and feature deep dives for v1.37
***
👇 Click to Read Full Raw Transcript
Swapnil Bhartiya: Let’s talk about some of the heavy hitters. If you look at scale to zero for horizontal pod autoscaler, it is graduating to beta and is on by default in this release. Can you talk about what does that unlock for teams running idle batch queue or GPU workloads and how big is the real cost impact here?
Dipesh Rawat: It’s a very good question. So it is one of my favorite features that has gone to beta in this release along with the other features that we have. So how it will save cost saving is typically without this feature you would have one pod running even without any demand that watches for some event driven triggers or some external metrics to scale up the workload. Now with this features being enabled so there is no need to keep a resource running to monitor those. So with this you can hook up to external metrics such as some event driven or some queue depth. So based on the demand, as soon as the demand comes up the workload automatically sets up. Now this will save you cost because there is no need to run those external monitors that is going to do your job for you. So it is the out of box feature that is being provided and it’s being enabled now so that it has graduated to beta. So I think it is going to save a lot of clusters and GPUs especially in the times where there is shortage of RAM and memory and there is lot of challenges in getting hold of those. So all in all I think this feature is a very good feature for people who are running AI workloads on their cluster.
Swapnil Bhartiya: Another major update is that resilient Watch cache initialization is now stable. What failure mode does that close during API server startup and recovery? And why does it matter more as clusters get bigger and bigger?
Dipesh Rawat: This is again one of my favorite features. So once the API server recovers or it comes back so it reboots, it rebuilds its cache. So during that time if the cluster is very large, which most of the production clusters are, clients are going to connect to those and they are going to get timeout. Now with resilient watch cache there is a way for the API server to reject those requests and give up appropriate HTTP response so that the clients can retry it while the API server is coming up. So all in all I think for a larger cluster this stable feature is going to enable them to have a more resilient way or more way of handling the outage or the maintenance I would say instead of taking a hit and making that outage window longer.
Swapnil Bhartiya: Now another of your favorite feature Pod label, checkpoint and restore just landed in alpha. What does that unlock for AI training jobs that get interrupted or need to migrate midrun and how far is it from something to teams can actually rely on in production versus just nice to
Dipesh Rawat: have I think this is one of the interesting features and one of the new features that have been introduced in this release. Now how this helps in AI workloads. So most of the AI training workloads or AI journal or modern workloads are very long running jobs. So if there is any disruption on those workloads. So traditionally what is going to happen? You need to run the workloads again. So if your job has run for two hours and it died at that 2R point, you have to again go back to point one and then again start from zero. Now what this features give you is it gives you a checkpoint. So at the point where the job goes down you can restart the job technically from there keeping all your progress that the job has done. So it is essentially going to help you speed up things where things go wrong due to disruptions in the cluster or disruption in due to some resources or some outage. Now for AI workloads this is going to help and to answer your other questions at how fast we can get it to beta and graduate. I think it depends on the feedback that we get and the graduation criteria that we have set up for this feature. So it needs to pass few criteria that it is production ready from our side of things that we do justice to our end users. But we highly recommend that turn on this feature, try to use this feature and please give us feedback that if it can be improved further to improve your use case.
