By The components inside AI server chassis have changed faster than the tooling built to manage them. SmartNICs, DPUs, specialized storage, distributed memory buses, GPUs, and power management subsystems now ship as standard in AI infrastructure, and most OEM management platforms do not reach all of them. At $200,000 to $250,000 or more per system, every hour a server spends on a loading dock or in a misconfigured state is a direct financial loss enterprises cannot absorb.
In this interview on TFiR, Rob Hirschfeld, CEO at RackN, breaks down why bare metal management has become dramatically more complex with the new generation of AI server chassis, why OEM brand loyalty no longer closes the gap, and how the accelerating split between training and inference workloads is reshaping hardware decisions.
Guest: Rob Hirschfeld, CEO at RackN
Show: TFiR
Here is what every platform engineer and infrastructure architect needs to know.
Technical Deep Dive
Q: Why are new AI server chassis harder to manage than previous server generations?
Rob Hirschfeld, CEO at RackN, explains that servers were relatively stable for roughly a decade, with Redfish providing some standardization but remaining heavily OEM-specific. The new generation of AI systems introduces SmartNICs, DPUs, specialized and tuned shared storage, distributed memory buses, GPUs, and power management components, each requiring its own configuration tooling. That tooling sits outside the reach of the OEM management platforms enterprises have relied on, which means teams can no longer assume a single vendor’s stack covers the full box.
“People who are really using AI tools heavily have to be able to manage the whole box and all of the components in the box, and they’re not going to get the lift that they might be used to from any of their previous generation tools.”
Rob Hirschfeld, CEO, RackN
Q: Can enterprises rely on a single OEM like Dell to manage their AI server infrastructure?
Hirschfeld is explicit: even if you are a Dell shop, you cannot count on Dell to handle all management and automation for AI-class systems. The new components require tooling that OEM platforms do not yet cover. Beyond tooling gaps, supply chain pressure and hardware availability make strict brand loyalty increasingly difficult to maintain, as enterprises may not be able to source the specific servers, RAM configurations, or components they need from a single vendor.
“If you’re a Dell shop, you can’t count on Dell to do all that type of management and automation.”
Rob Hirschfeld, CEO, RackN
Q: What is the real cost of slow provisioning on AI servers priced at $200,000 or more?
Hirschfeld frames the financial stakes directly. At $200,000 to $250,000 per system, and with that figure representing the low end of current AI server pricing, any time spent idle on a loading dock, in configuration, or in patching is an unacceptable operational loss. Supply chain delays have already compressed the window between order and deployment. Once hardware arrives, teams cannot afford additional delays from inadequate tooling or manual processes.
“The time pressure and time scale on these systems is just remarkably compressed.”
Rob Hirschfeld, CEO, RackN
Q: Is inference hardware splitting from training hardware, and does it change what enterprises need to buy?
Hirschfeld confirms the split is real and accelerating: inference has overtaken training as the dominant workload in 2024 and the trend is expected to continue. Purpose-built inference chips are arriving and will reduce some GPU demand at the margin, but they do not replace CPUs, RAM, or storage, and they do not fundamentally change the heat, power, or performance requirements of the surrounding infrastructure. Enterprises running real workloads, including open models at scale, still need high-RAM systems with GPUs or inference-specific chips alongside the rest of the stack.
“I wouldn’t look for anybody to say we’re going to have an alternative to Nvidia GPUs. That really isn’t going to change very much about what you have to deal with.”
Rob Hirschfeld, CEO, RackN
Q: Will inference-specific silicon from companies like Broadcom displace Nvidia GPUs for enterprise AI workloads?
Hirschfeld does not see inference silicon as a displacement event for Nvidia. The new chips will absorb some inference demand and may soften GPU requirements at the margins, but the broader infrastructure requirements, including GPU presence, high-capacity RAM, and the surrounding power and thermal envelope, remain. Models are getting larger, not smaller, for enterprises doing serious work, which sustains demand for powerful GPU-equipped systems regardless of what inference-specific chips add to the mix.
“Models are getting larger for companies that want to do real work. Even with open models, you’re talking about very powerful systems with a lot of RAM running GPUs or some type of inferencing chip.”
Rob Hirschfeld, CEO, RackN
Resources & Documentation
- RackN, bare metal lifecycle automation and multi-vendor infrastructure management platform
- Redfish (DMTF), standard API for out-of-band server management, referenced as the prior generation baseline
***
👇 Click to Read Full Raw Transcript
Swapnil Bhartiya: Now let’s talk about this new generation of integrated AI server chassis. Hardware is hard one spec changes and suddenly your whole stack doesn’t fit. So talk about what’s really different about these new systems and what should enterprises watch for to separate real innovation from the hype.
Rob Hirschfeld: So there is a tremendous amount of innovation going on in the hardware space. There’s the AI use cases the AI leaders are really driving new chassis designs, new components, new capabilities, new performance requirements and then all of the subsequent drivers configuration changes, capabilities on top of that. So when we look at servers today, servers have been relatively stable for the last decade or so, with the biggest innovation being Redfish coming in to provide some degree of standardization but, but still very OEM specific. The generation of tooling that we’re seeing now with SmartNics or DPUs, with specialized storage and tuned shared storage and memory, distributed buses, definitely GPUs, and now more and more power management, all of those capabilities are driving new components in the systems, new configuration tooling. That tooling is not reached by the OEM tools that you might be used to using. And so what we’re finding is people who are really using AI tools heavily have to be able to manage the whole box and all of the components in the box and they’re not going to get the lift that they might be used to from any of their previous generation tools. So you can, if you’re a Dell shop, say you can’t count on Dell to do all that type of management and automation. And what we’re finding is because of supply chain and availability, it’s much harder to stay brand loyal and get the benefits of trying to have brand loyalty that’s going to create situations where you might not be able to buy the servers you want or with the RAM or the configuration you need. And so all of these forces have come together to make the landscape of bare metal infrastructure significantly more challenging to even do the basics well, let alone really operate at scale. Like know if you’re spending, you know, 200, $250,000 and that’s probably the low end for some of these AI systems on a server. Then you know, you can’t afford to let it sit on the loading dock. You can’t afford to let it sit, you know, while you’re configuring it. You can’t afford to let it sit while you’re, you know, patching and updating it, those, those activities with that price of server. It has to get in the field, it has to get in use as soon as it possibly can. And we’re not even talking about the delays in acquiring the gear. It’s getting harder and harder to the infrastructure to show up on your loading dock. So the time pressure and time scale on these systems is just remarkably compressed.
Swapnil Bhartiya: on the hardware side. Training and inferencing have traditionally run on the same class of chips, but we are now seeing purpose built inference silicon like OpenAI’s work with Broadcom. Do you see training and inferencing? The hardware should fully diverge or will GPUs from Nvidia, AMD and other keep dominating both workloads?
Rob Hirschfeld: One of the things that’s happened this year is that the split between hardware for training and the hardware for inferencing, inferencing has taken over as the dominant workload and we expect that to continue to accelerate. The challenge that people need to reflect on is we will have a generation of inference specific chips. That’s not going to replace CPUs, it’s not going to replace RAM, it’s not going to replace storage. It might impact GPUs and lessen a little bit of the demand on GPUs. But fundamentally models are getting larger for companies that want to do real work. Even with open models, you’re talking about very powerful systems with a lot of RAM running GPUs or some type of inferencing chip. And so the components and capabilities around that infrastructure is pretty constant. So I wouldn’t look for anybody to say, you know what, we’re going to have an alternative to Nvidia GPUs. That really isn’t going to change very much about what you have to deal with. It’s not going to change the heat power performance. All of those requirements are all still bundled together.
