By Enterprises procuring GPU hardware for private AI clouds routinely discover that hardware acquisition is the trivial step. Achieving proper tenant isolation, fault tolerance, high-performance networking, accurate metering, and security with small, generalist infrastructure teams is where projects stall. GPU utilization rates of 15 to 20 percent are common, meaning the majority of capital investment sits idle.
In this interview on TFiR, Richard Borenstein, SVP of Growth and Business Development at Mirantis, walks through the three categories of risk enterprises face when building AI infrastructure without specialist support, and what operational capabilities are required to close the gap.
Guest: Richard Borenstein, SVP of Growth and Business Development at Mirantis
Show: TFiR
Here is what every platform engineer and AI infrastructure team needs to know.
Technical Deep Dive
Q: What are the main risk categories enterprises face when building AI infrastructure in-house?
Richard Borenstein, SVP of Growth and Business Development at Mirantis, identifies three buckets of risk. The first is tech stack complexity, which includes achieving proper tenant isolation, fault tolerance, and multi-tenancy requirements. The second is the depth of expertise required for GPU configuration and partitioning, high-performance networking, and storage. The third is operational efficiency: the ability to build and run this environment with a small team while maximizing return on investment.
“Buying the hardware is the easy part. Operationalizing it is the harder part.” — Richard Borenstein, SVP of Growth and Business Development, Mirantis
Q: Why is tenant isolation so difficult to achieve in private AI cloud builds?
Borenstein explains that building a private cloud requires proper tenant isolation and fault tolerance as the foundational layer, and that this challenge alone is significant. Small teams working across multiple projects simultaneously struggle to achieve even this baseline requirement. Without it, multi-tenancy requirements cannot be addressed safely, which compounds risk for every workload running on the platform.
“Building a cloud requires first and foremost proper tenant isolation and fault tolerance. That’s a challenge in and of itself.” — Richard Borenstein, SVP of Growth and Business Development, Mirantis
Q: What specific technical expertise do most enterprise teams lack for GPU infrastructure?
Borenstein points to virtualization, GPU configuration and partitioning, high-performance networking, and storage as the domains where most enterprise teams lack the necessary depth. These are not gaps that generalist infrastructure teams can close quickly. The combination of these requirements is what separates a functional GPU deployment from one that delivers a hyperscaler-like experience.
“Where some companies are challenged is in terms of expertise for virtualization and GPU configuration and partitioning, high performance networking, storage.” — Richard Borenstein, SVP of Growth and Business Development, Mirantis
Q: What does a hyperscaler-like private AI cloud experience actually require?
According to Borenstein, enterprises aspire to deliver the same experience their developers get from public hyperscalers. Achieving that on-premises requires a seamless, low-latency, high-performance environment with a cloud console, APIs, software and value-added services, accurate metering and billing, security, and proper multi-tenancy. Delivering all of these capabilities together is where in-house builds consistently fall short.
“What the companies are aspiring to is a hyperscaler-like experience: seamless, low latency, high performance, with the basics of a cloud console and APIs and software and value added services and accurate metering and billing and security and proper multi tenancy.” — Richard Borenstein, SVP of Growth and Business Development, Mirantis
Q: Why is GPU utilization so low in enterprise AI infrastructure deployments?
Borenstein reports that utilization rates of 15 to 20 percent are common across enterprise GPU deployments. This means the majority of capital investment in AI compute is sitting idle. Without advanced workload scheduling and placement, most of the hardware capacity that enterprises paid for is not generating value.
“A lot of these instances are not being utilized to their full potential. We’re seeing 15, 20% usage.” — Richard Borenstein, SVP of Growth and Business Development, Mirantis
Q: How do bin packing and workload placement improve GPU utilization?
Borenstein identifies bin packing and workload placement, combined with multi-tenancy, as the primary mechanisms for driving GPU utilization higher. These techniques allow multiple workloads to share physical infrastructure more efficiently, which maximizes return on investment from the hardware. Without them, GPU clusters remain underutilized regardless of the demand that exists across the organization.
“We’re helping companies to optimize and utilize through bin packing and workload placement and multi tenancy.” — Richard Borenstein, SVP of Growth and Business Development, Mirantis
Q: What does operational efficiency look like for a private AI cloud, and why does team size matter?
Borenstein states that enterprises need to build and operate their AI cloud environment with a relatively small team, or the economics do not work. Operational efficiency also requires a degree of optionality and configurability that allows teams to offer differentiated services without significant additional effort. When the operational burden exceeds what the team can sustain, utilization and ROI both suffer.
“They need to be able to build and operate this type of environment with a relatively small team or else the math doesn’t work.” — Richard Borenstein, SVP of Growth and Business Development, Mirantis
Resources & Documentation
- Mirantis, managed AI and cloud infrastructure platform for enterprises building private GPU clouds
***
👇 Click to Read Full Raw Transcript
Swapnil Bhartiya: We have been talking about, of course doing it yourself leads to complexity, all those challenges. But can you talk about what are some of the risks that enterprises may be running into when they do try to tackle it themselves, when they do try to solve the AI infrastructure complexity themselves without taking support from the whole ecosystem?
Richard Borenstein: Yeah, I mean it’s really kind of three buckets if you will. In my mind it’s like the tech stack complexity is meaningful. It’s really intense and intricate. And building a cloud requires first and foremost proper tenant isolation and fault tolerance. That’s a challenge in and of itself. I think with small teams working on multiple projects it’s hard to achieve even that. But addressing multi tenancy requirements is also vital and that means we can bring to the table. And where some companies are challenged in terms of expertise for virtualization and GPU configuration and partitioning, high performance networking, storage, like the reality is that buying the hardware is the easy part, operationalizing it is the harder part. And as we were saying earlier, like most teams just don’t have that experience today. And that’s where we can come in because ultimately what they’re hoping for is like a hyperscaler like experience. That’s what the companies are aspiring to. And a seamless, low latency, high performance environment with the basics of a cloud console and APIs and software and value added services and accurate metering and billing and of course security and proper multi tenancy. It’s difficult to do all of that and those are real challenges for our customers. And then lastly operational efficiency, like they need to be able to build and operate this type of environment with a relatively small team or else the math doesn’t work. Maximize the return on investment by driving the highest possible utilization. A lot of these instances are not being utilized to their full potential. We’re seeing 15, 20% usage and we’re helping companies to optimize and utilize through bin packing and workload placement and multi tenancy that I mentioned. And they want to have a degree of optionality and configurability to afford differentiated services without much effort. So these are the things that the problems we’ve identified and what we’ve tried to solve for them. Those challenges sound like the right ones that you know, for companies that you were thinking about swap.
