Securing AI Inference in Untrusted Infrastructure | Anand Kashyap, Fortanix | TFiR

Swapnil Bhartiya: Now, when it comes to enterprises, we all rely on encryption. But encryption protects data at rest and in transit. But the moment an AI model runs, inference model weights and data flowing through them sit decrypted in memory, which means a privileged admin or a cloud operator can potentially reach them. And as more of that AI is now moving into containers, only Kubernetes, that gap actually follows it into production. CNCF has confidential Containers project that brings hardware based trusted execution environments to Kubernetes. And Fortnix is now extending its confidential AI platform to run AI workloads inside them and to unpack that all. Today we have with us Anand Kashyap, co founder and CEO of Fortune Protonix. Anand, first of all, it’s great to have you on the show.

Anand Kashyap: Thank you, Satna, Nice to be here.

Swapnil Bhartiya: It’s my pleasure. As before we started recording or talking how old the company is, and you said it’s like roughly 10 years old, which means the company predates all these AI that we have been seeing recently. But you folks also kind of immersed in the whole container Kubernetes era. I want to hear from you, first of all, talk a bit about the company and then before we talk about Kubernetes, all these confidential computing, all the AI workloads, how you have seen the ecosystem and the workloads evolve, because you have seen that journey.

Anand Kashyap: So we started the company back in 2016, which was pre AI and at that time the thesis was that in the hardware in the CPU there were some new capabilities, security capabilities that were being built, which can secure your data in use using encryption. So just like you, you use encryption to secure data at rest and you use encryption to secure data in motion. Now you could use these instructions in the CPU to secure data in use. And later on that technology was called confidential computing. It provides two very important properties. It provides confidentiality or privacy of your data and IP or your code. And it provides integrity protection. So you can trust the software that is being run and you can trust the hardware on which it is running. And the combination of confidentiality, integrity and the whole confidential computing technology has allowed data to be protected for any kind of data and general purpose data. And one use case that we came upon was around encryption and key management. So we built a key management system built on confidential computing. We call it Data Security Manager. It’s a unified hsm, KMS Secrets manager. All of that combined into one single product and one single solution. And that was an early use case for confidential computing. We have demonstrated how powerful it is how it can secure data inside an organization. Regulatory industries like banking and healthcare, as applications have become more powerful and the confidential computing technology has become more available and more powerful as well. Now it’s available across all the CPU vendors, whether it’s intel or AMD. It’s available also in accelerated computing with Nvidia GPUs. And now it has enabled us to run an entire AI system, for example, an inference server inside confidential computing and protect the model weights as well as the data it’s processing at runtime.

Swapnil Bhartiya: Thank you so much. Of course, when you look at config the whole consortium, you folks were the founding members and now there is also containers. Let’s just focus on containers. Let’s just focus on AI. Why does confidential containers matter? And who feels the pain the most without it?

Anand Kashyap: Today, if you think about any application, there are multiple ways to run an application. On a bare metal machine, you could just be running an application binary directly on bare metal. Nobody does that these days since virtualization came along. Or you could run a VM and run your application inside the VM on a hypervisor. That is still popular and people do that. But more often than not, especially when you’re running AI and you’re running it for either for inference or you’re running agents, the preferred way of running those applications is inside a container. And as an application developer, you build a container image and that creates a portable package of your AI application. You can run it on any kubernetes based system, whether it’s native kubernetes managed kubernetes from any of the vendors. And it becomes very easy to now run AI, scale it up and down as needed. When more demand comes in, you want to spin up more container instances and serve the traffic. And you can scale it down when the demand is low. So containers have become the preferred way to run AI. And in the open source community, confidential containers or the COCO project, has become an important construct by which people are building containers and deploying it in confidential computing.

Swapnil Bhartiya: Now can you walk me through what actually happens to data and model width during inference when it runs on infrastructure that we cannot fully trust and control?

Anand Kashyap: Yeah. So confidential containers create a packaging for an application when the application now runs inside a confidential computing environment or a trusted execution environment. And it could be a combination of CPU as well as gpu. And you’d have CPU loading the application, loading the container, but then the model weights get loaded in the GPU memory and inferences done by using the GPU compute resources as all of this is happening. What configuration computing allows you to do is create a memory area where the model weights as well as the data get loaded. And that memory region, whether it’s a CPU memory or the GPU memory, they stay encrypted at all times. So anybody who is trying to either do a wiretap or try to dump the memory of the system, what they will get is just garbage because it’s all encrypted data. These are encrypted with keys that are burned and tied in the CPU and in the gpu. What Nvidia and their CPU partners have also done is make sure that the wire, the bus on which the data travels between CPU and gpu, that stays encrypted as well. And if you have a multi GPU system, even then all the GPU to GPU communication is encrypted as well. So now you get a completely encrypted system where the data is not visible from anybody sitting outside or even inside. Even if you have root permission, you still cannot look into the data. And while this is good for securing data and model weights at runtime, what you can also do is extend this trust boundary all the way to where the models were initially created, then they were encrypted, and make sure that the encrypted weights are now packaged inside the container image and then they are distributed. So now when the application comes up, there is a process called attestation by which you can verify that it’s the right application running on the right machine, the right hardware, under the right configuration. And all of this can be measured and validated by the hardware. And only then the encryption key which was initially used to encrypt the model weights are released directly inside this container, the confirmation container, and then the model weights get decrypted, then the model runs, and at no point of time the model weight is exposed to a third party, to a malicious insider or somebody who’s trying to attack the system. It creates kind of an end to end encrypted loop by which models are distributed and they’re unsecurely inside a confidential computing environment.

Swapnil Bhartiya: Now I also want to talk a bit about both sides. Like if you look at the sensitive data going in and valuable model IP running on Kubernetes, how do enterprises protects both side at once?

Anand Kashyap: Yeah, so when it comes to deployment of AI in the enterprise, I like to think of it as there’s sometimes three interested parties. There are model builders who are building their model. They spend a lot of time, effort, money to train these models, and for them, the model weights are extremely important. They don’t want their model weights to be compromised or stolen or changed when they’re running. Then there’s the enterprise who’s actually using an AI model. And a lot of times they prefer proprietary models. They want to have the model as close to their data as possible. They want to run the models in their on prem environment. They don’t. They’re not very comfortable, especially with the most sensitive data, to send it out over the Internet. Then there is an infrastructure provider. Now, in some cases, the infrastructure could be owned by the model builder itself. We see that with the big AI labs like OpenAI and Anthropic. Sometimes the infrastructure could be with the enterprise when they’re trying to run something on prem. But there’s also a case where the infrastructure could be a third party. This could be a NEO cloud or a sovereign AI provider as well. So now we have three parties. We have a model builder, a model owner, we have an enterprise, which is the data owner, and then we have an infrastructure provider. And there is a lack of trust between each of these three parties and none of them trust each other. The model builder wants to protect their model weights, the enterprise wants to protect their data. The infrastructure has to show and demonstrate that everybody else can trust them as well. What confidential computing does is it provides the mechanism for all these three parties to establish trust between each other. If you can put the model and the data inside a confidential computing, you can attest to it. You can make sure that encrypted data can only be decrypted after attestation. And the keys can come from FIPS 143, Level 3 validated HSM inside the secure enclave. Only then the data gets, the model weights get decrypted, only then the model runs. Now you have a trust system that you have created, which is based on technology, based on hardware, rather than what the parties are saying to each other or what contracts they’re signing. So it’s more about proof and verification of both software and hardware, rather than promises made between the three parties.

Swapnil Bhartiya: You just mentioned earlier attestation. Can you talk about how does attestation work in this setup and what has to be verified before a single encryption key gets released.

Anand Kashyap: Yeah. So the way Fortanics takes advantage of the attestation capabilities of confidential computing is we build a small piece of software called attestation client that we embed inside a container image. Now, as the container gets deployed on a target system, that the station client comes up as the container starts and it measures the hardware, which is both the CPU as well as the gpu. It measures the configuration in which the hardware is. It measures the software as well. So it measures the hash of the binary of the application running inside the container image or the model weights. It packages all of that and creates a certificate signing request for a private key that was generated inside this trusted execution environment. All of that is sent to our cloud service, which is called Confidential Computing Manager, or CCM for short. Now, CCM has connections to attestation services from Nvidia, from intel, from AMD, or whoever else is providing confidential computing hardware. So now CCM can verify that the attestation report being sent by the attestation client, can that be trusted? Can that be. Is it actually a genuine attestation report being created by a genuine piece of hardware? The second thing it does is can the application image be trusted? Is the application image the same compared to what it was when it was created and packaged by the application developer, for example, a model builder? When they’re distributing secure models, they would first measure the model weights and add those measurements inside the CCM system. So now CCM will check whether the model weights are the same compared to what they were packaged inside the container image. And the hardware can be trusted. The hardware configuration is correct, the BIOS has been updated, it’s running the latest microcode, and if all of that checks out, then we say, okay, now the attestation is successful. The cpu, the gpu, the application, they can all be trusted. And then CCM issues a certificate to the application. Now, the application can use this certificate to authenticate itself to our key management product, which is called Data security manager or DSM. It’s a unified HSM and KMS. It’s validated for FIPS 143, Level 3, which means it’s most secure place to store and manage your encryption keys. When DSM gets a request to release a key, it checks whether the request is coming with a certificate that has been issued by ccm. There’s a prior relationship that established between DSM and CCM and only after everything is verified is a key released from DSM to inside this container, which is running. And with this key, the key could be used to either decrypt encrypted model weights, or it could be used to decrypt a persistent storage where some data will be written onto the disk. There could be multiple keys as well. So this is a mechanism by which you do attestation and then based on that attestation, you release keys or secrets to the application.

Swapnil Bhartiya: Excellent. Once again, thank you. Now, if I’m not wrong, a lot of your customers are in regulated industry, like banks and governors. How does this change the conversation when they want to run Frontier models next to their own data?

Anand Kashyap: Yeah. So if you look at banks, healthcare industry, government, which are primary target and primary industry, which is most interested in confidential computing, what they’re worried about is they don’t want their data to be exposed, they don’t want their data to leak and go somewhere where they lose control of what the data is. Right. If a bank is building some kind of credit risk software, they’re worried that their data could be used and maybe distributed with some competitor banks as well. So increasingly what we’re seeing is that the banks, healthcare, government agencies, they’re asking the Frontier labs and the leading edge AI labs, can we take your model weights and just run it on our own? We don’t want to send all our data out to your cloud instance because we don’t know how it’s going out, what path it takes, where it can be intercepted. We have seen recent news about the controversy, for example, with the Navier Strokes math problem, where there was a lot of controversy around whether the work that the mathematicians were doing did that influence the work that OpenAI did to solve the math problem. Right. So in absence of certainty from the Frontier Labs saying, yeah, we will not look at your data, will not retain your data, will not train your models with their data. Enterprises want more control. They want to be able to run the best models. And they don’t want just open weight models, they do want the Frontier class models and they want to run it in their own controlled environment, preferably on prem. And as they’re trying to do that, there needs to be a way by which the model weights are protected so that the Frontier Labs can feel that, yeah, my model weights are not getting exposed. At the same time, the data of the enterprise, that needs to be protected as well. So confidential computing allows us to do that. And with our confidential AI solution built with confidential containers, now, it makes it much more easier.

Swapnil Bhartiya: Now, let’s just talk about the actual users from their perspective. What does day two looks like? Deployment upgrades, key management at cluster scale, once Confidential AI is running in production.

Anand Kashyap: Yeah. So typically there could be a single party, two parties or three parties which are involved in building out a confidential AI solution. As I mentioned, there could be the enterprise itself, which is making sure that their hardware is equipped with confidential computing. We deploy a piece of software called Node agent on every hardware that makes sure that the hardware is up to date, it can be attested to it, and it is equipped to run confidential computing software. The hardware is right. Then there is the model builder who when they build their models, they need to do a couple of things as well. So using the Fortanics control plane, they need to create encryption keys, they need to encrypt the model weights, they need to package the containers with the encrypted model weights, and then they need to register the measurements with our control plane to make sure that at runtime the measurements match. And then there’s the infrastructure provider in some cases, and they need to make sure that they integrate with our control plane and make it easy for an enterprise and for the model builder to be able to either run the model or distribute the model. So all these parties need to do some work. What binds everything together is our control plane, which essentially consists of two distinct products. One is the Photanics DSM or data Security Manager for managing keys. And the other one is Protonix Confidential Computing Manager which is used for attestation, verification, issuing certificates and managing the security policy.

Swapnil Bhartiya: From what I know you folks are building these, of course, you’re also part of Confidential Computing Consortium as open source plugin to coco. Why contribute upstream instead of kind of, you know, just, just do something? For particular, what value do you see in open source there?

Anand Kashyap: Open source software drives faster adoption. You know, confidential computing has been around for 10 years and while we at Fortimix have been preaching the use of confidential computing for a long time, there have been a lot of challenges as well. The availability was limited, the awareness was limited as well. And then the complexity of adoption was also high. It was not very easy for anybody who doesn’t know much about confidential computing to just take advantage of that and run their workload in confidential computing. What the Cocoa project is doing is it’s making it much more accessible for any regular enterprise, any regular developer, to not just build an application, containerize it and just deploy it in confidential computing, right? So we see that the open source movement towards confidential containers to be a tremendous boost in terms of adoption of confidential computing technology. Now, just running an application in computational container is not enough. If you’re not doing attestation securely, if you’re not managing the security policy around it, if you’re not doing secure key release in the right way, if you’re not managing your keys in an hsm, the solution is incomplete. So that’s where we come in. We provide the enterprise grade Integration for the Cocoa project. So for an application developer, for an enterprise it becomes super easy to build their applications and deploy them. But at the same time we provide a very easy low touch integration to manage the attestation, manage secure key release. They get full enterprise grade auditing, single sign on enterprise grade UI where they can go and visualize and have a dashboard to see what applications they’re running on what infrastructure.

Swapnil Bhartiya: One last question before we wrap this up is that coco let’s talk about the maturity of the or production readiness of the project. COCO reached incubation and the fact is that incubation or graduation at Kubernetes it doesn’t mean the project not ready as matured, it’s already being used in production. What kind of gaps the project still have that enterprises might hit and they should be aware of?

Anand Kashyap: Look, Cocoa is a relatively new open source project. The world of confidential computing has traditionally been quite fragmented. There are hardware available from various vendors, Intel, AMD, Nvidia, Fujitsu, bringing confidential computing CPUs as well. Soon the various ways to deploy these applications and the open source project has been maturing. The Cocoa project has been maturing. It didn’t have GPU support earlier, it only has cpu. But since then it has been added I think right now. And then there are some corner cases as well, right? In a multi GPU system, how does COCO work versus not so these things will continue to get better and continue to improve, right? And the support for various configurations of CPU and GPU CPU only ARM based CPUs and things like that will continue to get better. Our objective has been to be the partner with the open source community, provide the enterprise grade attestation and security release which will be needed regardless of what the hardware configuration is. So I think today Botanic support is ready. It’s available in public preview. People can use us to go deploy their applications with COCO and with Photonics control plane. As support for various hardware configurations and confidential computing continue to be added in Cocoa, they will automatically be supported by Photanics as well going forward.

Swapnil Bhartiya: Anand, thank you so much for of course talking about confidential containers and Photanics and where the market is heading. It’s. It was a great discussion, thank you so much and of course for our viewers please go and check more about [email protected] and back to you Anand, thank you for your time and I look forward to chat with you again.

Anand Kashyap: Thank you, thank you very much.