By 
The expensive GPU you aren’t using
A multi-node inference job is ready to run. Some of its pods get scheduled; the rest wait for capacity. The pods that landed hold GPUs, but the job can’t serve a request. When another replica finally starts, it loads the same model weights from storage that its neighbors already loaded.
GPU capacity is costly and often hard to find. Wasting it while a job waits to start is painful. So is discovering that the capacity you found on another cloud requires a different collection of drivers, node settings and deployment configuration before you can use it.
Kubernetes gives us a common way to describe workloads, but it doesn’t make GPU environments identical. Each provider has its own taints, network setup, storage classes and driver lifecycle. Moving an inference workload therefore involves more than moving a manifest. The team has to bring its operating assumptions along, then work out which ones still hold.
The next constraint on AI inference isn’t only how many GPUs a team can get. It’s whether it can use the capacity it has, wherever that capacity happens to be.
Every cloud has its own trap
Capacity might be available on a hyperscaler for one deployment and a GPU-focused cloud for the next. Platform teams often handle the differences by copying a working configuration and adapting it for each provider: one set of Helm values for EKS, another for GKE, and more for AKS, OCI and CoreWeave.
That’s a reasonable way to get the second environment running. By the fifth, the copies have histories of their own. A driver update reaches one cloud but misses another. An annotation appears in two places, with no explanation of why the third lacks it. It’s hard to tell which differences are required and which are accidents.
Then someone asks a simple question: “Which GPU driver version is configured for production on GKE?” The answer may be buried under charts, values and overlays. To find it, an engineer has to render the configuration or ask the live cluster what it is running. Those are different questions. The rendered configuration shows intended state; checking the cluster shows observed state. A useful platform should make both clear, without confusing one for the other.
When a team can’t readily inspect its intended configuration across environments, portability wears away one fork at a time. Lock-in shows up through configuration rather than through a contract.
Separate intent from provider details
The useful pattern is to describe what the workload needs, then resolve the provider details from a shared set of rules. For GPU configuration, those rules might select components by service, accelerator, operating system, intended use and platform. They should produce the deployment bundle for each environment and leave the differences visible.
That has two benefits. A platform engineer can compare the rendered outputs before deployment and see exactly what changes between providers. An SRE or auditor can query those outputs later to learn which versions were selected for each environment, without first getting access to every cluster. Observed state still requires observation, but the team no longer has to reverse-engineer intended state from a pile of templates.
The same separation matters to ML teams. They should be able to declare a model and its serving requirements without embedding a particular cloud’s taints or annotations in that declaration. The platform remains responsible for mapping the request to compatible capacity.
An open source fleet control plane is one way to put that division into practice. Modelplane, built on the CNCF graduated Crossplane project, manages inference deployments across clusters and providers. Its ModelDeployment gives the ML team a stable interface while the platform manages placement and the infrastructure underneath. That doesn’t erase provider differences. It puts them where platform engineers can manage them.
Portability takes the whole stack
A fleet control plane can choose where a model should run. It cannot, by itself, make a multi-node engine start together or load weights efficiently inside a cluster. Those jobs belong to the serving stack. If the two layers don’t cooperate, a portable deployment can still leave expensive GPUs waiting.
Recent work between Modelplane and NVIDIA offers a concrete example. Modelplane can compose a serving stack based on NVIDIA Dynamo. Grove and the KAI Scheduler help place a multi-node engine as a group, so part of the engine doesn’t claim GPUs while the rest waits. Where the network supports it, ModelExpress can transfer model weights from a peer’s GPU instead of making each new replica read them from storage. A storage path remains available when peer transfer isn’t possible.
There’s a configuration lesson here, too. NVIDIA AI Cluster Runtime (AICR) publishes recipes for compatible combinations of GPU drivers, operators and related software that NVIDIA and the AICR community test and validates. Modelplane takes its driver and operator versions, along with node-tuning and monitoring configuration, from AICR recipes when building each cloud’s serving stack. Rather than asking every platform team to choose each component version independently, the stack starts from a known combination. Validation reduces guesswork; teams still need to check that a recipe fits their hardware and environment.
A cluster’s choice of serving stack is fixed, but a fleet can run different stacks side by side. Teams can add a Dynamo-based cluster and move deployments gradually. The ML team’s ModelDeployment stays the same. That’s a more useful test of portability than whether two clouds happen to accept the same YAML: can the platform change underneath a workload without making the ML team rewrite its contract?
Make portability something you can inspect
There are a few questions worth asking of any cross-cloud GPU platform. Can you render the configuration for each target before applying it? Can you compare those results and explain every provider-specific difference? Can you query the selected driver and operator versions without touching a cluster? And can you separately verify what is actually running?
At KubeCon + CloudNativeCon North America, we’ll show the configuration side of that pattern in One Recipe, Five Clouds, One Query. We’ll use a shared recipe library to render GPU configuration for five environments, compare the results to show what carries across and what changes, then query the rendered configuration without cluster access.
No recipe makes five clouds identical. It can make their differences legible. For teams deciding where to place scarce GPU capacity, that’s the first step toward making portability an operating practice rather than a promise.
KubeCon + CloudNativeCon is taking place in Salt Lake City from November 9 to 12. Register Now!
