Swapnil Bhartiya: When it comes to AI workloads, they don’t behave like typical applications. Here GPUs sit idle and burn money, clusters grow and the control plane struggles to keep pace when it comes to training jobs. They run for days and one interruption can erase all that progress. Now Kubernetes 1.37 is going to tackle all of these problems, cutting idle resource cost, hardening the the API server, and laying the groundwork for workloads to survive interruptions instead of starting over again. And today we have with us Dipesh Rawat Release lead For Kubernetes version 1.37 at CNCF to walk us through all that’s new in this release. Dipesh, great to have you on the show.

Dipesh Rawat: Thanks for having me on the show.

Swapnil Bhartiya: Let’s talk about some of the heavy hitters. If you look at scale to zero for horizontal pod autoscaler it it is graduating to beta and is on by default in this release. Can you talk about what does that unlock for teams running idle, batch queue or GPU workloads and how big is the real cost impact here?

Dipesh Rawat: It’s a very good question. So it is one of my favorite features that has gone to beta in this release along with the other features that we have. So how it will save cost saving is typically without this feature you would have one pod running even without any demand that watches for some event driven triggers or some external metrics to scale up the workload. Now with this features being enabled so there is no need to keep a resource running to monitor those. So with this you can hook up to external metrics such as some event driven or some queue depth. So based on the demand as soon as the demand comes up the workload automatically sets up. Now this will save you cost because there is no need to run those external monitors that is going to do a job for you. So it is the out of box feature that is being provided and it’s being enabled now so that it has graduated to beta. So I think it is going to save a lot of clusters and GPUs especially in the times where there is shortage of RAM and memory and there is a lot of challenges in getting hold of those. So all in all I think this feature is a very good feature for people who are running AI workloads on their cluster.

Swapnil Bhartiya: Another major update is that Resilient Watch cache initialization is now stable. What failure mode does that close during API server startup and recovery and why does it matter more as clusters get bigger and bigger?

Dipesh Rawat: This is again one of my Favorite features. So once the API server recovers or it comes back so it reboots, it rebuilds its cache. So during that time if the cluster is very large, which most of the production clusters are, clients are going to connect to those and they are going to get timeout. Now with resilient Watch cache there is a way for the API server to reject those requests and give up appropriate HTTP response so that the clients can retry it while the API server is coming up. So all in all, I think for a larger cluster, this stable feature is going to enable them to have a more resilient way or more way of handling the outage or the maintenance I would say instead of taking a hit and making that outage window longer.

Swapnil Bhartiya: Now another of your favorite feature, Pod level Checkpoint and restore just landed in alpha. What does that unlock for AI training jobs that get interrupted or need to migrate Midrun and how far is it from something teams can actually rely on in production versus just nice to have.

Dipesh Rawat: I think this is one of the interesting features and one of the new features that have been introduced in this release. Now how this helps in AI workloads. So most of the AI training workloads are AI journal or modern workloads are very long running jobs. So if there is any disruption on those workloads. So traditionally what is going to happen you need to run the workloads again. So if your job has run for two hours and it died at that two hour point, you have to again go back to point one and then again start from zero. Now what this features give you is it gives you a checkpoint. So at the point where the job goes down you can restart the job technically from there keeping all your progress that the job has done. So it is essentially going to help you speed up things where things go wrong due to disruptions in the cluster or disruption in due to some resources or some outage. Now for AI workloads this is going to help and to answer your other questions that how fast we can get it to beta and graduate. I think it depends on the feedback that we get and the graduation criteria that we have set up for this feature. So it needs to pass few criteria that it is production ready from our side of things that we do just is to our end users. But we highly recommend that turn on this feature try to use this feature and please give us feedback that if it can be improved further to improve your use case.

Swapnil Bhartiya: Now if you put all of these three together vastcacher Resiliency HPA scale to zero and of Course checkpoint restore. What does that tell us about where Kubernetes is heading, where the focus is? It’s quite obvious. AI is the biggest workload these days. Though initially people used to say that AI is just another workload on Kubernetes or Linux. But no, AI is changing, changing everybody’s, you know. So talk a bit about what does it tell about Kubernetes and its focus.

Dipesh Rawat: Just like you said, that all those features are giving us a hint as to where the Kubernetes is going in the future releases. So what I would like to say it’s kind of a coincidence that the features that we are building is also helping the AI community. And also we are taking inputs from the community to bring those features such as dra, to have specialized hardware in the cluster. How do you request those and how do you can manage those? So I would say that more such features are in the pipeline and the features that are already in alpha and beta, based on their graduation criteria, we are looking to graduate them to stable so that they are more production ready and can be used widely by the teams. So I would say that all in all, Kubernetes as a whole, we are trying to make it workload aware. So instead of you giving us a pod and we just taking that resource, declarative resource and just scheduling it, Kubernetes on the whole is trying to be workload aware. What type of workload is it? Does it require some type of new hardware? If that hardware is not present, what do we do? How do we manage that hardware? So all in all I would say yes, it is going to help AI workloads. And at the same time we are also making the control plane more resilient and more secure so that if long running and big jobs are running so the cluster does not go down because of that.

Swapnil Bhartiya: Now, as much as we are excited about all these new features, are there anything that are being deprecated in this release that users should be aware of?

Dipesh Rawat: We have a few deprecations that were part of the release. So first I would go with a quick one. So in kubectl, everyone’s favorite cli, we have deprecated the minus F or file name flag for kubectl run. So traditionally kubectl run was meant to run pods or resources directly from the flag. It was never meant to run a declarative spec that you gave it a YAML and it is going to run so that for that you use create and apply. So we have drilled down and made it More obvious, that F or file name should not be used by kubectl run command so that flag has been removed for kubect. IPVS mode is going to be deprecated in the future releases to IPVS mode was originally added for performance for IP tables and stuff. Now we have NF tables and we have got new features that are present with that. So we have decided to retire that mode. So currently if you start Kube Proxy in IPVS mode in the logs you will get that. You will get a message that it is going to be deprecated in the future releases. As said in the release announcement, it is going to be removed. Again, it’s not an immediate removal. We are giving you the deprecated warning according to our deprecated policy. So please try to migrate. If you are running Kube proxy in IPVS mode in your cluster, there is a particular command that you can use to identify what mode your KUBE proxy is running. The other deprecation is Kubedns is being deprecated. So for a long time CoreDNS has been the default DNS provider in our cluster. So Kubedns, we have decided as a project today to retire that project. So it is going to stop receiving security updates in the future. So if for any reason you are running kubedns, please try to migrate to newer core DNS. It will give you access to newer features such as Endpoint slices and in place memory QoS. So it’s a win win for everybody.

Swapnil Bhartiya: Now every release has name and there is a story behind the name. This release is named Garwal. Talk a bit about the story behind the name and how does it reflect the community, the contribution model behind this release.

Dipesh Rawat: So the code name for this release is Garwal. So Garwal is a region in India, in the northern parts of India in a state called Uttarakhand. So it’ region where I belong to. So the theme name of the release came from what my grandmother used to say to me, like everybody’s grandmother, that no matter how big or how successful you become in life, don’t ever forget your roots or where you come from. So that’s where I got the idea to honor my roots and maybe raise more awareness about that particular region in the country. So how I linked it to the release theme is so Garhwal is a mountainous region in Himalaya. So once I say Garwal, the typical image that comes into people’s mind is mountains, landscapes, or if you are more adventurous into hiking or trekking or seeing glaciers. But for me, there are people living in the rural villages, especially one of the villages where I come from. So it’s a very difficult terrain to live in, but people still live there and they help each other. And based on the knowledge that they have gained over generations, they try to pass it on and they try to build a communal feeling. And so it felt very much as similar to what we do in Kubernetes community as we come from all different aspects of life and then we come together for one particular goal and help to build Algota. For that matter, it resonates with every open source community that we have. So that’s the core theme. And then in the theme I have various elements from my region. So you can see a main Himalayan mountain, then there are the other forest, that is the state tree. Then you have a monal word, which is a very beautiful word. And then we have the state flowers, then we have the river flowing. So one Easter egg as a part of this logo is traditionally the release logos are static images. So I wanted to do something different. So if you see the release announcement and if you slow down and look at it for I think 37 seconds. Why 37 seconds? Because 1.37 the logo is going to switch. So it is at the start, it is a daytime logo. So after 37 seconds the landscape is going to turn into nighttime and you can see the stars. And just for a Easter eggs, the stars twinkling are one of the message that I have put in the logo. So if someone can figure out the stars are twinkling and saying what messages, please do tag me if you can figure out what that Easter egg is before teams.

Swapnil Bhartiya: You know, if you go back to some of these features, mostly AI focused, before teams turn any of this on, what are the things that they should evaluate first, how they should prepare themselves so there are no surprises and they get the most out of this release.

Dipesh Rawat: So first I would say is please go through the detailed release notes that we publish, especially look at the deprecation and migration guides that we have given. Verify in your cluster if you are running anything that is going to be deprecated in the new future and try to read the migration guides that we have provided now. Second, in terms of feature enabling, so please look at the features, especially the new features like Alpha features. So we are reliant on. We do have graduation criteria for those features, but we are reliant on your feedback also, if there are certain ways that that feature can be improved by changing the implementation, please try to enable those and look at your use case and come back to us raise issues either on GitHub or reach to that particular sig via Slack. That helps. And I would say try as many features as possible. I think beta features are generally safe to work with, but please I would say look at your cluster and look at your scenario if it is going to create any issues before running into it.

Swapnil Bhartiya: Of course we talked a lot about a lot of enhancement. Most of those were your favorite. But is there any other enhancement in this release that you think deserves more attention and we have not talked about it.

Dipesh Rawat: I would say this is a controversial gap, so I would say metrics KH IO API finally going to stable after being 9 years in beta should get the attention. Now there are questions as to why it was in nine years. So I would like to say that once that API was introduced in beta it predates our enhancement process mode. So before we had any sort of formal process of proposing features into kubernetes like the KEP process so that API predates that feature. So so I think it got forgotten a bit. So what we did is so despite it being beta for years, so people have been using in production and it has been fairly stable. So at this release it’s just formally acknowledging that beta API being stable and we are labeling it as sorry, we are labeling that beta into a stable API so nothing has changed underneath. So it is just a proper graduation of that beta API into stable API acknowledging that it has been running into production environments for lot of year and it needs to be graduated to stable. So I would say that is one of the caps that is more interesting to me in this release. And I would say that even though we are transitioning that API from beta to stable, I know people have been running that beta API far more years in production so the beta API is going to stick around during this transition period. But I would say that please try to see if you can migrate to beta APIs and don’t wait for last minute updates.

Swapnil Bhartiya: Dipesh I want to say that, but I would say this is one of my favorite release discussions on kubernetes so far. We have had great guests but this is I think one of my favorites. So thank you for taking your time out and explain things not only in that detail in depth but also make it so easy so people can actually understand the value that is bringing to them. And also thanks for telling a great story about this release, the whole Easter eggs there. So thank you so much and I look forward to chat with you folks again. Thank you.

Dipesh Rawat: Thanks for having me.