By Local GPU workstations running open weight models give enterprise teams a fast start, but they provide no audit trail, no governance controls, and no guaranteed availability for the agentic systems that production AI requires. As inference hardware diversifies beyond GPUs and AI agents demand always-on infrastructure, the gap between a developer laptop deployment and a production-grade AI platform is widening fast.
In this interview on TFiR, Rob Hirschfeld, CEO at RackN, covers why local model deployments are a short-term tactic and not a long-term strategy, what dedicated AI infrastructure actually requires at the hardware and platform layer, and how enterprises should build those skills in parallel rather than waiting for sequential decisions to resolve.
Guest: Rob Hirschfeld, CEO at RackN
Show: TFiR
Here is what every platform engineer and enterprise infrastructure team needs to know.
Technical Deep Dive
Q: Why are local open weight models on developer workstations a reasonable starting point for enterprise AI?
Rob Hirschfeld, CEO at RackN, explains that local models give individual end users, starting with developers, a way to leverage AI while reducing processing load on central infrastructure. Running models locally is fast to deploy, requires no centralized provisioning, and lets teams experiment without large upfront commitments. The value is immediate: developers can move quickly and explore open weight models without waiting for enterprise infrastructure decisions to resolve.
“Running local models and empowering developers is great, but from an enterprise perspective, you need to be planning further out.” — Rob Hirschfeld, CEO, RackN
Q: What are the specific enterprise risks of relying on local workstation AI long term?
Hirschfeld identifies three core failure modes: auditability, governance, and availability. When models run on individual machines, the enterprise has no reliable way to audit what models are being used, verify outputs, or enforce approved model policies. Availability is tied entirely to whether a specific user’s machine is running, which makes SLA guarantees impossible. Cost control also breaks down because usage is invisible and decentralized.
“It’s going to be very hard to audit and control and check when you’re limited to what that end user is doing on their machine.” — Rob Hirschfeld, CEO, RackN
Q: How does a centralized AI cluster improve cost control and governance compared to distributed workstation deployments?
A cluster of AI resources gives enterprises centralized visibility into utilization, which directly enables cost controls that are impossible to enforce across individual desktops. Hirschfeld notes that centralized infrastructure also allows teams to enforce approved model lists, manage which models are in production, and build governance controls around access and outputs. The cluster becomes the single plane of control for all AI consumption across the organization.
“Having a cluster of AI resources gives you better cost controls, but it also gives you better auditability and governance and lets you manage those pieces.” — Rob Hirschfeld, CEO, RackN
Q: Does enterprise AI inference require GPUs, or are there other hardware options?
Hirschfeld is direct: inference does not necessarily require a GPU. A new generation of inference-specific chips is entering the market, and modern CPUs are gaining dedicated inference capabilities. This means enterprises that lock their AI strategy to whatever GPU hardware exists on developer desktops are constraining themselves unnecessarily. A heterogeneous hardware mix allows teams to match specific models to the most appropriate compute, and to adopt new chip types as they become available without redesigning the entire platform.
“Locking yourself into what’s on people’s desktops is a good short-term solution, but long term, having that dedicated AI capability with a heterogeneous mix of hardware so you can match models to capabilities is essential.” — Rob Hirschfeld, CEO, RackN
Q: Why do agentic AI systems specifically require dedicated corporate infrastructure rather than developer workstations?
Agentic systems are not interactive tools that a user runs on demand. They need to operate continuously, run batch processes, and function in offline or low-latency contexts. Hirschfeld points out that agents running on developer laptops are subject to the same availability, audit, and governance failures as any other workstation-based deployment, but the operational consequences are worse because agents are often running automated workflows. Dedicated infrastructure provides the reliability and control surface that agentic workloads require.
“You don’t want a room full of cracked lid laptops providing your agents. You actually want them running in controlled corporate infrastructure.” — Rob Hirschfeld, CEO, RackN
Q: Why is an SLA for AI model access a critical enterprise requirement?
Hirschfeld notes that AI downtime or degraded performance stops work, and that dependency is only growing as more processes integrate AI. Guaranteed access to models, even non-frontier open weight models, is essential for running routine processes reliably. Without an SLA backed by dedicated infrastructure, enterprises have no recourse when model availability fluctuates. The harness layer that RackN provides is designed to deliver that SLA guarantee across heterogeneous hardware.
“Having guaranteed access to models to run processes is absolutely essential because more routine processes can be run very effectively by open weight models.” — Rob Hirschfeld, CEO, RackN
Q: What does the path from local AI models to owned enterprise AI infrastructure actually look like?
Hirschfeld pushes back against the common enterprise instinct to sequence every decision before acting. The path should involve parallel workstreams, not a linear progression. Developer-facing harness adoption can begin immediately. At the same time, hardware and operations teams should be learning how to spec, procure, wire, and run AI gear, and how to build Kubernetes clusters optimized for AI workloads. Waiting for one decision to resolve before starting the next is the primary cause of delay.
“You need to be building these skills in multiple dimensions simultaneously.” — Rob Hirschfeld, CEO, RackN
Q: How does the VMware migration experience apply to AI infrastructure planning?
Hirschfeld draws a direct parallel: enterprises dealing with VMware migrations tend to map out their final destination first and then work backwards, which causes them to underinvest in the intermediate operational capabilities they will need regardless of where they end up. The same pattern appears in AI infrastructure planning. Every possible AI destination requires better automation, better operational control of infrastructure, and stronger bare metal processes. Building those capabilities now is not premature. It is prerequisite work that applies universally.
“All answers require you to have better operational control of your infrastructure, more automation, better processes. This is what RackN does from a bare metal perspective.” — Rob Hirschfeld, CEO, RackN
Q: What is the biggest strategic mistake enterprises make when planning AI infrastructure?
Hirschfeld identifies the core mistake as deferring infrastructure investment until after major decisions about models and vendors are resolved. Enterprises tell themselves they will start building AI infrastructure once they know which LLM is best or which harness to use. But because AI innovation is moving so fast, that waiting period burns exactly the time that teams should be using to build operational skills in hardware, platform, and automation. Companies that build the platform first and then innovate on top of it consistently move faster than those who optimize the destination before building the road.
“Don’t wait on building AI infrastructure until after you’ve made those decisions. Start building all the skills across your teams so that they’re delivering the pieces that you need right out of the gate.” — Rob Hirschfeld, CEO, RackN
Resources & Documentation
- RackN, bare metal automation and AI infrastructure platform for enterprise on-premise deployments
***
👇 Click to Read Full Raw Transcript
Swapnil Bhartiya: Many enterprises are starting by putting GPUs in developer workstations and running local open weight models, which is a great way to get started. But why is that a reasonable place to begin but not a complete long term strategy?
Rob Hirschfeld: Well, there’s a couple of reasons why not, and one of them is innovation and pace of innovation. So I want to get there and put a pin on that and how you do it. But from an enterprise perspective, ultimately it’s going to be about control and audibility. So what we’re looking at here is if you’re running an enterprise system and you want to have your AI, your individual end users, you said developers, but it could be any end user leveraging a local model that is going to save you on processing capabilities. However, what you’re going to do is be limited to what that end user is doing on their machine, when their machine is running, when you have access to it, and it’s going to be very hard to audit and control and check. So the idea of having a cluster of AI resources actually gives you better cost controls, but it also gives you better audibility and governance lets you manage those pieces. And this is where the other dimension comes in. Inference does not necessarily require a gpu. There’s a lot of new chips coming into market that will provide AI inference capability and even running CPUs. And CPUs are getting new inference capabilities. So locking yourself into what’s on people’s desktops, it’s a good short term solution, but long term, having that dedicated AI capability with a heterogeneous mix of hardware so you can match models to capabilities and bring in new models whenever you need to, and then control to make sure that the models that you’ve approved are getting used. Those control points are absolutely essential and really from an enterprise perspective, critical to scale. And it’s worth noting when you look at how those pieces go. And that’s exactly what we’re helping companies spec and build today. The harness allows you to pull those pieces in, you guarantee an sla, right? If you’ve been. We get caught up in AI, goes down, has a day where it’s not performing well and a lot of work gets stopped. You are going to want to be able to control even if you’re not on a frontier model. Having guaranteed access to models to run processes is absolutely essential because more routine processes can be run very effectively by open weights models. And along those lines, what you also need to think about is in that infrastructure and with those controls. And there’s one more critical point with this, which is that it’s not just your users, your developers and people doing this. Part of what you need to be thinking about building is agentic systems. And those agentic systems have to run somewhere. Also, you don’t want a room full of cracked lid laptops providing your agents. You actually want them running in controlled corporate infrastructure. And so all of these things work together to drive this realization that you want to have a dedicated AI infrastructure that can run your inference models in a controlled way, allow your agents to work especially offline, or to provide batch operations in a controlled way. So all of these pieces fit together. So as much as I love running local models and empowering developers from an enterprise perspective, you need to be planning further out.
Swapnil Bhartiya: And now, what does the path from that first step to owned AI infrastructure actually looks like for an enterprise?
Rob Hirschfeld: Wow, this is one of those ones. And actually it’s funny because VMware migration is similar to this in our books and experience is enterprises have a tendency to try and sequence out all these steps and they think they have to do this in a very orderly manner. Our experience actually is that the path should have a lot of parallel operations. You know immediately, right, that you can empower developers to do local models and look at harness. So harness is clearly a first step. But the people who are going to make an evaluation for your harness, right, your platform team, your developers, aren’t the same ones who actually are the ones who should be looking at how to run and spec AI gear and infrastructure that’s mostly hardware teams and operation teams that need to understand how to set up and run AI gear, how to build kubernetes from an AI workload perspective. And what I would encourage, especially because AI innovation is running so quickly, is that you need to be building these skills in multiple dimensions simultaneously. So you should be looking on how do I improve my delivery of AI infrastructure? How do I know what to buy? How do I know how to wire it together? How do I build these systems and get that skill set embedded in your organization? We see exactly the same thing going on with a lot of VMware choices, where people try to figure out what their exit out of VMware should look like and then work backwards. The reality is, all answers require you to have better operational control of your infrastructure, more automation, better processes. This is what Rackn does from a bare metal perspective. What we’ve seen is that companies who build the platform and the capabilities that allow them to then innovate on top of that platform end up moving a lot faster than the ones who figure out where they have to get to at the end of the trip. And don’t worry about, you know, any of the intermediate skills they’re going to need to build, especially in today’s market. You need to be making sure you’re jumping through all the intermediate skills simultaneously to build this up. So don’t get tied up in I need to figure out my hardest. I have to figure out which LM is best. I have to. You do need to do those things, but don’t wait on I’m going to be building AI infrastructure until after you’ve made those decisions, start building all the skills across your teams so that they’re delivering the pieces that you need right out of the gate.
