By Enterprise AI hardware strategies were built for model training: tightly coupled clusters, fast fiber optic interconnects, and homogeneous node configurations. Inference workloads operate under a different set of constraints, and applying training-era assumptions to inference deployments is generating real performance bottlenecks and stranded capacity. The primary chokepoint in a shared inference system is often not the GPU at all, but how context windows are stored, loaded, and fed into the model across concurrent users.
In this interview on TFiR, Rob Hirschfeld, CEO at RackN, walks through why inference infrastructure demands heterogeneous hardware sourcing, what the CPU-to-GPU balance actually means for system design, and why the old model of buying identical machines and differentiating at the software layer no longer holds for AI workloads.
Guest: Rob Hirschfeld, CEO at RackN
Show: TFiR
Here is what every platform engineer and AI infrastructure architect needs to know.
Technical Deep Dive
Q: How does inference infrastructure differ from training infrastructure?
Rob Hirschfeld, CEO at RackN, explains that training infrastructure was optimized for fast interconnects, specific storage arrays, and tightly coupled large clusters, and those requirements remain valid for that use case. Inference, however, is structured around user interactions with a model, where every prior exchange in a session forms a context window that must be loaded, stored, and fed into the model continuously. The performance demands are real but architecturally distinct from what training requires.
“The bottleneck in a lot of cases is how you manage and hold that context window.” — Rob Hirschfeld, CEO, RackN
Q: Why is context window management the primary bottleneck in AI inference systems?
In an inference system, the context window represents the entire history of a user’s interaction with the model, and that data must be held in RAM and fed into the model in a specific sequence. Hirschfeld notes that in shared inference environments, the system must load and swap context windows between different users as they each make requests, which places significant and continuous pressure on memory capacity and memory bandwidth. This makes RAM architecture, not raw GPU compute, the dominant design constraint for shared inference deployments.
“As you build up a big system, especially a shared inferencing system, you actually have to load the context in between different users as they feed into those systems.” — Rob Hirschfeld, CEO, RackN
Q: What role does CPU-to-GPU balance play in inference performance?
Hirschfeld points out that inference workloads can become bottlenecked on the CPU rather than the GPU, which is a failure mode that catches teams off guard when they assume GPU capacity is always the limiting resource. The balance between CPU processing, GPU compute, and available RAM determines where throughput collapses under load. Getting that ratio wrong means expensive GPU capacity sits underutilized while a weaker CPU component throttles the entire pipeline.
“Sometimes you can get bottlenecked not on the GPU, but on the CPU behind this.” — Rob Hirschfeld, CEO, RackN
Q: When should enterprises consider ASICs or special processors for inference workloads?
Hirschfeld explains that certain models and certain workloads are better served by purpose-built processors, whether ASICs or other specialized silicon, than by general-purpose GPUs. The decision depends on the specific model being run and the nature of the inference task. When a team is running the same model at consistent scale, optimizing the hardware to that specific workload through purpose-built silicon becomes worth evaluating against the cost of general GPU capacity.
“If they have special ASICs or special processors to run certain models, what you’re going to end up looking to do is optimize that mix.” — Rob Hirschfeld, CEO, RackN
Q: Why are heterogeneous hardware clusters replacing homogeneous clusters for AI inference?
Hirschfeld draws a direct contrast with traditional enterprise IT, where organizations bought identical machines, wired them as a cluster, and differentiated at the VM or software layer. For AI inference, different models and different workloads have different optimal hardware profiles in terms of RAM, CPU-to-GPU ratio, and processor type. Unless an organization is running identical models at massive uniform scale, forcing all workloads onto homogeneous hardware results in stranded capacity across the fleet.
“Unless you’re running the same model and the same workload at incredibly large scale, you’re going to have a much more heterogeneous mix of systems, or you’re going to end up wasting a lot of capacity.” — Rob Hirschfeld, CEO, RackN
Q: What is a small language model and why does it matter for enterprise inference cost?
Hirschfeld describes small language models as models trained from a larger foundation model to become a subject matter expert on a narrower domain, which reduces the overall model size. For enterprises, a smaller model requires less RAM, less GPU memory capacity, and imposes lighter context window management overhead, all of which reduce the per-query cost of running inference. He identifies this as the path large enterprises will take as they look to make inference workloads economically sustainable at scale.
“Training a smaller model based on a larger model to become a subject matter expert on one topic makes the model smaller.” — Rob Hirschfeld, CEO, RackN
Q: Why does AI hardware scarcity make capacity waste an urgent problem?
Hirschfeld notes that AI hardware remains difficult to procure, which means organizations cannot afford to buy excess capacity to compensate for architectural inefficiency the way they might in a more abundant hardware market. Wasting capacity on hardware that is mismatched to a workload is a direct financial penalty in an environment where getting additional units is not straightforward. Hirschfeld does not expect that supply constraint to ease in the foreseeable future, which makes getting the hardware mix right a near-term priority rather than a future optimization.
“Hardware is hard enough to get that you can’t afford to waste capacity, and I don’t see that changing in the foreseeable future.” — Rob Hirschfeld, CEO, RackN
Resources & Documentation
- RackN, bare metal automation and infrastructure lifecycle management platform for AI and cloud-native deployments
***
👇 Click to Read Full Raw Transcript
Swapnil Bhartiya: Now let’s talk about inferencing. Why does inference infrastructure change the conversation around multi vendor sourcing and even previous generation hardware?
Rob Hirschfeld: So, and this is where things get a little tricky. We did spend a lot of time about two years ago very focused on what model builders needed, which is very fast interconnect. It focused on having a certain type of storage array, having these large clusters that were very tightly interconnected. Those are still important, important. And I do think that if you’re a large enterprise, at some point you will look at doing small language models where you train a smaller model based on a larger model to become a subject matter expert on one topic, because it makes the model smaller. And this, I think is what people need to understand about these models from an inference perspective is the way the inferencing system works is your interaction with the model. Everything you’ve done so far is a context window. The bottleneck in a lot of cases is how you manage and hold that context window. From a storage perspective, that takes a lot of ram. It has to be loaded and fed into the model in a certain way. And as you build up a big system, especially a shared inferencing system, you actually have to load the context in between different users as they feed into those systems. And so that is actually one of the biggest performance, performance bottlenecks on how an inference system is built. Now, the nice thing is that you don’t need to chain together tons and tons of systems with fiber optic networking between them to support training the model, but being able to build and maintain these systems, to inject the context windows into a model, store the model on the GPUs or in memory, all of those things are all still very demanding. And so there’s an interesting balance between the, the size of the models you use, how you load the context windows, how big those context windows can be, how effective the processing can be. Sometimes you can get bottlenecked not on the gpu, but on the CPU behind this. And so what we’re seeing is there’s a lot of ways to do inferencing. We’re still working out what the most effective model and system is. And in all likelihood, depending on the model and the workload that you’re doing, you’re going to have different systems for different models and different use cases. And that’s really important. When we look at the cost of these systems and how much RAM they have and the CPU to GPU balance, or if they have special ASICs or special processors to run certain models, what you’re going to end up looking to do is optimize that mix. So the idea that you’re buying 64 machines wiring together as a VMware cluster and then the differentiation is in the VMs, as we used to see in traditional enterprise IT when we start looking at what a cluster would be for AI inference. Unless you’re super large scale and you’re running the same model and the same workload at incredibly large scale, it’s possible that you’re going to be having a much more heterogeneous mix of systems in your thing, or you’re going to end up wasting a lot of capacity. And frankly, hardware is hard enough to get that you can’t afford to waste capacity and or I don’t see that changing in the foreseeable future.
