As enterprises push generative AI into production, a new class of infrastructure challenges is emerging—particularly around how to efficiently serve large language models at scale. A new open source project, llm-d, is aiming to address that gap and has now been accepted into the Cloud Native Computing Foundation (CNCF) Sandbox.

The move brings llm-d under the CNCF’s open governance model, positioning it alongside other early-stage projects working to extend Kubernetes into new workload domains. In this case, the focus is on distributed AI inference—an area where traditional cloud-native patterns are still evolving.

Making AI Inference a First-Class Kubernetes Workload

Kubernetes has become the standard platform for orchestrating containerized applications, but AI inference introduces a different set of requirements. Unlike stateless microservices, inference workloads are highly stateful, latency-sensitive, and dependent on hardware characteristics such as GPU memory and cache locality.

This mismatch has led to inefficiencies in how AI workloads are scheduled and scaled. Conventional routing and autoscaling mechanisms are typically unaware of inference-specific factors like prompt size or token generation phases, resulting in inconsistent performance and underutilized resources.

llm-d is designed to address these limitations by introducing a Kubernetes-native framework for distributed inference. It sits between higher-level serving platforms such as KServe and lower-level inference engines like vLLM, aiming to bridge orchestration with execution.

By integrating more intelligence into routing and scheduling decisions, the project seeks to improve how inference workloads are placed and executed across clusters.

A Collaborative Push Toward Open AI Infrastructure

The project was launched in 2025 as a joint effort involving major players across the cloud and AI ecosystem, including Red Hat, Google Cloud, IBM Research, CoreWeave, and NVIDIA. It has since attracted contributions from additional vendors and research institutions, reflecting growing industry interest in standardizing AI infrastructure.

At its core, llm-d promotes a hardware-agnostic approach—supporting inference workloads across accelerators from multiple vendors. This aligns with a broader cloud-native principle of avoiding vendor lock-in, which is becoming increasingly relevant as organizations adopt diverse AI hardware strategies.

The project is also expected to align with CNCF-led initiatives such as AI conformance efforts, helping ensure interoperability across tools and platforms in the ecosystem.

Rethinking Traffic, Scaling, and State Management

One of llm-d’s primary contributions lies in making inference-aware decisions within Kubernetes. For example, it introduces routing mechanisms that consider model state and cache utilization, rather than relying solely on generic load-balancing techniques.

It also separates different phases of inference—such as prompt processing and token generation—into independently scalable components. This allows infrastructure teams to allocate resources more efficiently, addressing imbalances that can occur when both phases are tightly coupled.

Another area of focus is state management. The project includes capabilities for handling key-value (KV) cache data across multiple tiers, from GPUs and CPUs to storage systems. This is critical for optimizing performance in large-scale deployments where memory and latency constraints are tightly linked.

Additionally, llm-d leverages Kubernetes primitives to orchestrate complex, multi-node workloads, enabling distributed inference patterns that would otherwise require custom infrastructure.

Bridging Cloud Native and AI Ecosystems

The acceptance of llm-d into the CNCF Sandbox reflects a broader convergence between cloud-native technologies and AI systems. As enterprises operationalize AI, the need to integrate model serving with existing Kubernetes-based workflows is becoming more urgent.

Projects like llm-d aim to close that gap by providing standardized building blocks for AI infrastructure. They also highlight the growing role of open source in shaping how AI systems are deployed and managed.

The project’s contributors are also exploring closer collaboration with adjacent ecosystems, including machine learning frameworks and research communities, to create a more seamless pipeline from model development to production deployment.

What Comes Next

As llm-d enters the CNCF Sandbox, its future will depend on community adoption and real-world validation. The project’s emphasis on benchmarking and reproducibility suggests a focus on proving performance gains rather than relying on theoretical improvements.

If successful, llm-d could help define how distributed inference is handled in Kubernetes environments—bringing more predictability and efficiency to one of the most resource-intensive parts of the AI stack.

For enterprises, the takeaway is clear: as AI workloads become mainstream, the underlying infrastructure will need to evolve. Projects like llm-d represent early efforts to build that foundation, turning AI inference into a manageable, cloud-native capability rather than a specialized, siloed system.