By AI inference workloads running inside containers expose model weights and live inference data to the underlying host stack. Standard container isolation stops at the OS boundary. It does not protect data while the CPU is actively processing it. For regulated industries and multi-tenant cloud deployments, that gap is not theoretical. It is a compliance failure waiting to happen.
In this interview on TFiR, Anand Kashyap, CEO and Co-founder at Fortanix, walks through how confidential computing closes the in-use data gap for containerized AI workloads, and why the open source Confidential Containers (CoCo) project has become the architectural foundation teams are building on.
Guest: Anand Kashyap, CEO and Co-founder at Fortanix
Show: TFiR
Here is what every platform engineer and AI infrastructure team needs to know.
Technical Deep Dive
Q: What is confidential computing and how does it protect data in use?
Anand Kashyap, CEO and Co-founder at Fortanix, explains that confidential computing uses hardware instructions built directly into modern CPUs to encrypt data while it is being processed, not just when it is stored or in transit. The technology provides two core guarantees: confidentiality of data and code, and integrity verification of both the software running and the hardware it runs on. Fortanix has been building on these CPU-level trusted execution environment (TEE) capabilities since 2016, before the term confidential computing was widely adopted.
“Just like you use encryption to secure data at rest and you use encryption to secure data in motion, now you could use these instructions in the CPU to secure data in use.”
Anand Kashyap, CEO and Founder, Fortanix
Q: Why have containers become the default runtime for AI inference workloads?
Kashyap notes that while bare metal and VMs remain in use, containers have become the preferred deployment unit for AI because they package the application as a portable, portable image that runs consistently across any Kubernetes-based system, whether self-managed or a managed cloud service. The ability to scale container instances up and down dynamically in response to inference demand makes Kubernetes the natural orchestration layer for AI. That shift toward containers creates a direct need for a security model that extends hardware-level protection into the container runtime.
“Containers have become the preferred way to run AI, and whether you’re running it for inference or running agents, the preferred way is inside a container.”
Anand Kashyap, CEO and Founder, Fortanix
Q: What is the CoCo (Confidential Containers) project and who needs it?
The Confidential Containers (CoCo) open source project provides a standard construct for deploying containers inside confidential computing environments, giving Kubernetes-scheduled workloads the same hardware-rooted confidentiality and integrity guarantees previously available only to VMs. Kashyap identifies AI developers building inference servers and agent runtimes as the primary audience, along with organizations in regulated industries such as banking and healthcare that cannot accept in-use data exposure. Fortanix was a founding member of the CoCo project and continues to build on it as a core part of its platform.
“In the open source community, confidential containers, or the CoCo project, has become an important construct by which people are building containers and deploying in confidential computing.”
Anand Kashyap, CEO and Founder, Fortanix
Q: What does Fortanix Data Security Manager do and how does it relate to confidential computing?
Kashyap describes Fortanix Data Security Manager as a unified product that combines HSM (Hardware Security Module), KMS (Key Management System), and Secrets Manager functionality into a single solution, built from the ground up on confidential computing. It was one of the earliest production use cases demonstrating that confidential computing could handle real enterprise workloads, including key management for regulated industries. The unified approach eliminates the operational complexity of managing separate security infrastructure components.
“We built a key management system built on confidential computing. We call it Data Security Manager. It’s a unified HSM, KMS, Secrets Manager, all of that combined into one single product.”
Anand Kashyap, CEO and Founder, Fortanix
Q: Does confidential computing support Nvidia GPUs for AI acceleration?
Kashyap confirms that confidential computing has expanded beyond CPU vendors Intel and AMD to include Nvidia GPUs, which means the full AI inference stack, including the accelerated compute layer, can now run inside a trusted execution environment. This development makes it possible to protect model weights and inference data at runtime inside a complete AI system, including the GPU-accelerated inference server, not just the CPU-side control plane.
“It’s available also in accelerated computing with Nvidia GPUs, and now it has enabled us to run an entire AI system, for example an inference server inside confidential computing, and protect the model weights as well as the data it’s processing at runtime.”
Anand Kashyap, CEO and Founder, Fortanix
Resources & Documentation
- Fortanix, confidential computing platform including Data Security Manager (unified HSM, KMS, Secrets Manager)
- Confidential Containers (CoCo) Project, open source CNCF project for running containers inside trusted execution environments on Kubernetes
***
👇 Click to Read Full Raw Transcript
Swapnil Bhartiya: Talk a bit about the company. And then before we talk about all this confidential computing, all the AI workloads, how you have seen the ecosystem and the workloads evolve, because you have seen that journey.
Anand Kashyap: So we started the company back in 2016, which was pre AI, and at that time the thesis was that in the hardware, in the CPU there was some new capabilities, security capabilities that were being built, which can secure your data in use using encryption. So just like you use encryption to secure data at rest and you use encryption to secure data in motion, now you could use these instructions in the CPU to secure data in use. And later on that technology was called confidential computing. It provides two very important properties. It provides confidentiality or privacy of your data and IP or your code. And it provides integrity protection. So you can trust the software that is being run and you can trust the hardware on which it is running. And the combination of confidentiality, integrity and the whole confidential computing technology has allowed data to be protected for any kind of data, general purpose data. And one use case that we came upon was around encryption and key management. So we built a key management system built on confidential computing. We call it Data Security Manager. It’s a unified HSM, KMS, Secrets Manager, all of that combined into one single product and one single solution. And that was an early use case for confidential computing. We have demonstrated how powerful it is, how it can secure data inside an organization, regulatory industries like banking and healthcare, as applications have become more powerful and the confidential computing technology has become more available and more powerful as well. Now it’s available across all the CPU vendors, whether it’s Intel or AMD. It’s available also in accelerated computing with Nvidia GPUs. And now it has enabled us to run an entire AI system, for example, an inference server inside confidential computing, and protect the model weights as well as the data it’s processing at runtime.
Swapnil Bhartiya: Thank you so much. Of course, when you look at confident compute, the whole console shim, you folks were the founding members, you know. And now there is also for containers. Let’s just focus on containers. Let’s just focus on AI. Why does confidential containers matter? And who feels the pain the most without it? Today,
Anand Kashyap: if you think about any application, there are multiple ways to run an application on a bare metal machine. You could just be running an application binary directly on bare metal. Nobody does that these days since virtualization came along. Or you could run a VM and run your application inside the VM on a hypervisor that is still popular and people do that. But more often than not, especially when you’re running AI and you’re running it for either for inference or you’re running agents, the preferred way of running those applications is inside a container. And as an application developer, you build a container image and that creates a portable package of your AI application. You can run it on any Kubernetes based system, whether it’s native Kubernetes, managed Kubernetes from any of the vendors, and it becomes very easy to now run AI, scale it up and down as needed. When more demand comes in, you want to spin up more container instances and serve the traffic, and you can scale it down when the demand is low. So containers have become the preferred way to run AI. And in the open source community, confidential containers, or the CoCo project has become an important construct by which people are building containers and deploying it in confidential computing.
