Confidential Computing for AI on OpenStack | Goutham Pacha Ravi, OpenStack Community | TFiR

Swapnil Bhartiya: Now what is happening is that AI workloads, they are changing the very infrastructure that they run on. Teams need specialized compute, GPU accelerators and bare metal. But they also need to trust where those workloads actually run, especially when the data is sensitive or regulated or you work in a compliant industry. The OpenStack committee just shipped Hibicus, its 34th release with major advances in confidential computing security, AI and HPC infrastructure. And it’s not just about new features, it’s also about how the community builds and secure the software that the whole world relies on. And today we have with us Gautam Pacha Ravi, chair of the OpenStack technical committee. Gautam, it’s great to have you on the show.

Goutham Pacha Ravi: Thank you for having me.

Swapnil Bhartiya: It is my pleasure. If I’m not wrong, this release is framed as trusted infrastructure for the AI era. What does that mean in practice? Because trust could mean different things to different people. But let’s just focus on what does it mean in practice for teams building AI platforms on OpenStack.

Goutham Pacha Ravi: I want to preface that by saying that yes, this is our 34th release and we’ve been doing this for about 16 years as a software community and as an open source software community. So a lot of the trust is drawn upon that experience of shipping and building software in the open. So while the workloads changed and the use cases changed, the principles that are underlying this software have not changed. So that’s what that partially means. So AI is, yes, putting some new demands on this infrastructure, as you mentioned, and in the Hibiscus release, that’s where the community’s investment has gone. So we’ve hardened the security, the governance that produces this software. And so at the same time, it’s not just adding the capabilities that AI is demanding, but also the fact that we are shipping the software in the same way as we’ve always done in the open, and so that you can, as somebody that’s deploying the software, if you’re a security inclined, you should feel free to verify the claims that the software is making because it is open source. So you can read the code. These are not promises that a vendor is making to you that you have to take it at value without verification. So partially that’s what the trust means. And in this Hibiscus release, I think one of the things that we did was the capabilities themselves, unlocking capabilities for AI and HPC. That would mean that we’ve worked more on accelerators, we’ve worked more on bridging the storage gap. So it’s basically a lot of high performance storage work is happening, integration work is happening in this space. We have new protection mechanisms. So these are auditable, these are hardware enablement basically. So they are the ones that concern confidential computing. There is a lot of security discipline that’s also renewed in this cycle for us, so I can talk more about that. But basically it’s what is all encompassed in this trusted infrastructure that we’re probably touting this as to be. Yeah, so a lot of the work is disaggregated into a lot of the projects and we can get into specifics if you’d like. But the underlying point is that when you’re running a large cloud and you’re serving AI infrastructure, all of this can come together for you as auditable and securable infrastructure for you to be able to meet your regulatory requirements.

Swapnil Bhartiya: When you look at this release, of course confidential computing is a big theme this release with of course Nova adding support for AMD CV SNP and Intel TDX. What does that enable for organizations running sensitive or regulated workloads?

Goutham Pacha Ravi: So AMD SNP is something from the vendor. So it is a third generation secure, encrypted, virtual. So it basically means stronger integrity protection for your data and it’s coming with that platform attestation. And there’s the same thing with Intel TDX, it expands to trust domain extensions. So on 5th generation Xeon, scalable or later, this is a capability that vendor provides. So both of those are now possible to be exposed via OpenStack Nova. So as a self service API, so that’s where OpenStack shines. So you don’t as an operator or as an end user have to configure these things by hand or through arcane scripts. You can actually use standard OpenStack APIs to deploy this. And that new capability as it is unlocks different things in your cloud. One of the things is maybe because of the kind of data that you were handling and the kind of workloads that you’re running, you could not consider shared infrastructure for it before, but now you can, because you’ve gotten these capabilities that allow you to meet a regulatory bar. And so you can have a compliance officer inspect your deployments and see that the data that is getting handled, this data protection applies while it is in use. It’s not just data protection that’s in transit, data protection for data that’s at rest, it’s memory protection while it’s being used. So the data that you are processing on these virtual machines with these extensions are not visible to the host even. So nobody can actually get in and see this data. So this includes the operators that you’re renting or you’re leasing this shared infrastructure. So where does this make sense? Well, all the highly regulated industries, my favorite examples are healthcare and financial services. So with these, as you can tell, it’s private data that’s protected and it’s highly sensitive, that should anybody have access to it, they could be huge penalties to you as an organization. And also it’s a lot of trust that you have built from these users and you’re extending to your cloud service provider as well. So imagine now that you’re running your AI workloads on top of this, you’re training models with this highly sensitive data and you’re using these confidential computing mechanisms. This allows you to make sure that your model’s weights and the data itself that they process, they’re not exposed to anybody besides you.

Swapnil Bhartiya: Perfect. Awesome. Thank you. And of course, another core feature is our focus is also on security. This release also hardened how vulnerabilities are managed across the project from the vulnerability management team all the way to influx of security researcher. Why should that process matter to organization deciding whether to run its critical workload?

Goutham Pacha Ravi: So this is a bit of a story time. I think in the past release, as an open source community, we derive from what the community decides to work on and how the community determines work needs to be done. For the past over a decade, we have done vulnerability management in a particular way where we have allowed project teams to tell us that their threat models and to work with a centralized vulnerability management team to handle their work vulnerabilities. So anybody that reports a vulnerability interfaces with this vulnerability management team and they can then choose to responsibly disclose this vulnerability to the project team and then get a fix out who will then choose to actually get this through an embargoed process so that this responsible disclosure extends to the operators that are actually using this in critical systems. So the story time is that in the past release, at the fag end of it, we extended this VMT coverage to the whole OpenStack. So that’s something predictable we expect as end users and as operators, you expect that all of your software is being treated consistently like this. And that was the difference. Although with the way the community has evolved, this is what was needed to make that change. So when you report a vulnerability today to the VMT, the VMT triages it with core security practitioners from within each project team. And we have a standard protocol, a triage process. And we develop a fix if one is warranted. And once a fix is developed, we test our assumptions with the reporter and then we start looping in a few more folks at that point. And then we extend the fix in embargo to anybody that is running OpenStack in a critical manner. This could be package maintainers, deployers, operators, distributions and so on. Anybody that’s signed up to get an early warning of a vulnerability and a patch. And so these patches are going out to those folks and then within five to 10 business days from that, this is going into public disclosure. So this embargo process helps protect these sensitive places especially. So imagine financial institutions, healthcare like we discussed, or even research clouds and government clouds across the world. So they are patched even before an attacker could discover this vulnerability and actually take advantage of it. That’s the hope and that’s the streamlined process that changed and actually made Hibiscus a bit more secure than prior releases. I would be lying to you if I said this is done. Security isn’t a feature that we ship just once and we forget this is a constant practice and this is a discipline that the community is putting together and it’s taking on. So Hibiscus is probably the beginning. And every time we do one of these vulnerability disclosures, you will notice that we handle all of the older stable releases that we maintain. And that’s for the benefit of operators that are, as we all know, are not running the latest greatest software. And yeah, that’s the central aspects of security for this release.

Swapnil Bhartiya: Of course OpenStack is established, but what makes OpenStack a credible foundation for AI infrastructure versus relying on, let’s say, a hyperscaler.

Goutham Pacha Ravi: Thanks for that question. So hyperscalers bring in quite a lot of goodness. They take away quite a lot of your pain in managing infrastructure. But there is an inherent risk in that the risk is in control and in lock in. So hyperscalers are, at the end of the day, in the business of making sure that they can keep your applications and your data in one place. The gravity helps them, but that works against you when the landscape shifts. So the hyperscaler or even a proprietary vendor of some sort of software that you are using could end up changing the rules underneath you, or they could be forced to do so by the sovereign powers that they are subject to. And so that’s where open source software like this would be an answer. So it’s about making sure you have control. You have no lock in. There are no vendors that can control and hold you hostage, you have the flexibility to move things around and you have, you have the APIs locked down in air gapped environments and at the edge or any form factor that suits your business rather than having to tinker with what is capable in a hyperscaler cloud. And from this community that’s been around for this long, there’s also the aspect that you don’t really give away your control of the roadmap. So you get to participate in the community as an end user, as an operator and request features or contribute to them. And that way we are not catering to a particular corporate entity, whether that’s a vendor or distributor or anything of that sort. So there’s that. And then as AI infrastructure is growing, as AI use cases are growing, it’s a hardware bet. So you start thinking about three, five, so many years in your horizon and you want to make sure that your substrate that’s building on top of this hardware is solid for that. They’re not going to change the terms for you within that time frame. And so where your data is running, where the software is running, how much control do you have on it on the roadmap? That’s what motivates all of this and gives you a comparison to work with. This was already true of large research institutions, large telecommunication institutions, large service providers themselves. There was a bent towards away from hyperscalers because of the costs, because of the need for control, because of regulation, because of sovereignty requirements and so on. The AI is just increasing that demand in my view.

Swapnil Bhartiya: Now let’s also look at one of the pain points which is of course NVMe, RAM and storage, AI and HPC workloads are reshaping storage demands as well. In past, you know, if you remember, you know, like just store everything storage cheap. Not anymore. Storage is not even hard drives, you know, or I don’t know, the tapes anymore, how expensive even they are becoming more an expert. So now which is actually a good thing, you know, you cannot just collect all the data and then you’ll figure something out later. But AI and HPC workloads are reshaping storage demands as well. How is OpenStack keeping up with it, including the new version and lustre support in Manila?

Goutham Pacha Ravi: It’s a great point because as we’re seeing, I think if you look at the charts, I mean data is only growing exponentially and our storage demands are just scaling with that. As informative as that is, everything that we are producing has got to be stored somewhere and it should not be an afterthought. Where AI and HPC are making this more challenging is their demand, their hunger for faster storage, for easier access. And it shouldn’t be in the AI, you know, compute world to have to figure out how something flows between a compute and storage itself. And so that’s where I think the storage industry all the way back from the traditional storage vendors to the absolutely new modern technologies that are being built to support AI workloads are all trying to integrate into an OpenStack cloud. And that’s what we see in this latest release as well. So we had HPC capable storage always in OpenStack, exposed through various different kinds block object file. So whether that was Ceph or there was Vast Data, there was NetApp, there was different kinds of storage technologies, and now there is in this release a new modern storage, VekaFS. And we’ve added a driver for Lustre. So Lustre, as you know, is a file system that runs most of the world’s HPC workloads. It runs in national labs, supercomputers have relied on it for years. And so unlocking that capability allows OpenStack to be used in the same sense and for Lustre to be usable through OpenStack. So this could be for training your models for building out any kind of modern AI pipeline through this whole self service mechanism that OpenStack was always providing. So the parallel use cases are driving up more and more of these. So I expect to see improvements in this, whether that’s unlocking new capabilities and existing storage integrations or net new capabilities. I’m already seeing some on the horizon for the next release as well.

Swapnil Bhartiya: Can you talk about what role is AI playing in open stack code development while you’re leveraging AI? But AI is not making decisions, AI is not taking creativity. AI is more or less like doing all the heavy lifting. So if you can talk about the role of AI in your development process, that will help us a lot.

Goutham Pacha Ravi: That’s a great question and it’s a complicated one. So rather than picking sides, I think we’ve had a very pragmatic view from the get go. We are a community driven organization. Like, I mean it’s basically the community is the software. Because technology is something that’s getting more and more democratized. You could definitely build your own cloud today. You could build whatever software you would picture because LLMs are capable of actually helping you and do more with the kind of powers you unlock with them. So that’s great. But the idea of a community driven software project was not about the technology that was. It was also about the principles. And so it was. I mean, there were a lot of capabilities that you could have unlocked prior that we chose not to, because we know what that. I mean, we in our collective wisdom kind of understood what the tech landscape would be and what is better for folks that are using this software. So we’re able to connect with that use case and try to gather multiple viewpoints and actually build cohesive software despite all of that. So with that bent, I can tell you that we approach the AI assistants with the same kind of view. We said, okay, so this is an enabler. We are seeing this, but there are some downsides. Like you said, it is a very powerful technology that we can’t take for granted. And so if somebody were to bring in a lot of code changes, were to bring in things, what would our response be with AI assistance? And when I say somebody, it’s not like these folks are coming from outside of the community. Our community is free and open to join at any time. And we’re also talking about people that are going to do this from within the community. And so, I mean, I think early 2024, we started working on an AI policy with the OpenInfo foundation, that’s the foundation that supports the OpenStack project. And it was a very reasonable policy that we came up with. We were looking for folks to attest their use of LLMs when developing the code. And for us this was important because we wanted to make sure that they were doing the due diligence of making sure that they’re not landing code that is plain stripped off from somewhere else without doing a compatible license check and so on. This brought the onus on the contributors to say that are you able to, I mean, spare some time to think about the implication of what it is that the code is enabling and where the code comes from and so on. And we encourage the use of open source models and open weight models to kind of get to that same means as well, because we are not sure that all of these LLMs are pretty black box to us in terms of how their training is and so on. That said, we’ve evolved that AI policy now we have captured more aspects of it. We do believe that a human is still in the center of judgment for this. We’ve said that anybody that is using AI assistance, this is welcome. This is fine. Just be assured that we expect you to understand what it is that you’ve contributed and be aware that maintainers are going to ask you this so they would expect that you were able to explain the code, just as this was done even before the age of AI assisted contributions. So when you submit code, maintainers are signing up to understand and to maintain that code from here on forward. That includes you as a contributor. And so that contract is still not changed. So that’s been our response to this. But I can tell you that there have been great good sides to this, that a lot of the innovation that’s there in the past release has been AI assisted. So there is good usage of this to optimize things. There’s good usage of this to assist maintainers. There’s good usage of this to grow CI CD capabilities, things that we were putting off from testing. We just needed some more time and some more hands. And LLMs are giving you that capability without having to spend a lot of time as humans thinking through a particular problem or there’s a backlog of things as you’re working through. So this allows you to reduce that backlog a little bit. And we are seeing those efficiency gains. And that’s something that the community appreciates. Again, we are also, as maintainers, writing out recipes into our code itself, into the repositories. We’re crafting some skills and knowledge that folks can use to make their contributions more palatable and easier to onboard to OpenStack. So there is a positive embracing, if you ask me. But at the same time, the community is cautiously trying to avoid this situation where somebody treats the code as an entire black box and tries to say that, all right, I’ve got this fully coded project that I would like to bring in here. Take it, because my agent said it was right. So we’re fighting against that sort of a thing in a reasonable way. Because again, like I said, the central part of it is that human judgment has not left.

Swapnil Bhartiya: Of course, these days everybody is playing with AI and most of them, they start with hyperscalers. That’s how we start for teams that start on hyperscaler. And now they want to move their AI workloads because privacy. There are a lot of reasons they want to do that. They want to move their AI workloads onto their own infrastructure. What does that migration really look like?

Goutham Pacha Ravi: Yeah, so the primitives between the hyperscalers and OpenStack already exist because far, long ago, I mean, some people were trying to build out hyperscalers with OpenStack and there are still a lot of public cloud service provider aspects to OpenStack that. So that’s the flexibility in the product. So if you’re bringing in your AI workloads, this could be inference or AI pipelines of any sort. I mean, you’ll discover that many of the capabilities already exist and more often than not you’re actually layering infrastructure software and putting in container orchestration on top of it. And on top of that, are you running your MLOps workloads and stuff. And so this is where I think very recently the Open Infrastructure foundation and the Linux foundation folks had come together and put together a blueprint of how this would look like with pure open source software. And that seems like a good place to start for somebody that’s explored this with hyperscalers to see if all of your needs have already been met by a whole series of open source projects. And my bet is even with the hyperscalers, there’s a lot of open source innovation that you’re taking advantage of, except in this case, you’re in control of deploying it and managing it as well.

Swapnil Bhartiya: Gautam, thank you so much for joining me today and of course, talk about this release and those folks who are watching. If you want to learn more about OpenStack Hibiscus, head over to openstack.org and check it out. And of course, Gautam, back to you. It was a great discussion and I would love to have you back on the show. Thank you.

Goutham Pacha Ravi: Yeah, thank you, Swapnil. Take care. Bye.