Swapnil Bhartiya: Now, when it comes to AI, unlimited tokens sound great on paper, but enterprises are learning that more context does not automatically mean better outcomes. Without guardrails, reviews and clear operational controls, AI will drift. It will waste resources and produce unreliable results. That is where the idea of AI harness starts to make sense. RackN is focused on helping organizations bring structure to how AI is deployed and operated, especially as they move from experiments on local machines to real owned infrastructure. And today we have with us once again Rob Hirschfeld, CEO and co founder of RackN, to help us dig into what that transition really looks like. Rob, it’s great to have you on the show.

Rob Hirschfeld: It’s always a pleasure to be back and I love digging deep into these topics. So happy to share what we know.

Swapnil Bhartiya: With the whole backlash against token maxing, can you talk about what should enterprise executive understand about why unlimited tokens never translated to unlimited results? And where does human review fit into that?

Rob Hirschfeld: Oh my goodness. It really is funny to have watched people just decide they were going to handle engineers unlimited numbers of tokens and hope they would produce excellent results or even results, period. We use AI very deeply at RackN and I can tell you that after the initial euphoria, people start realizing just how carefully you have to manage and guide those processes, how much you have to watch what those outputs are. Because small decisions the AI make really snowball. And the challenge with unlimited tokens is that you have no feedback loop, you have no process controls in which to moderate the AI making a decision, thinking it’s right, and then driving very aggressively to build a lot of things around those failed assumptions. And AI really doesn’t have the intuition to know which assumptions are good or bad. And so the challenge with unlimited tokens, even generous token loads, is that you encourage people to build on top of bad assumptions. And in best case scenario you just end up rebuilding. And that’s great. If you catch the assumptions, you rebuild and you rebuild and you rebuild. It’s expensive. The worst case scenario is those assumptions end up embedded in your product, your product ends up shipped to the customer, business decisions get made about with those assumptions embedded in and then you’re trying to recover. And so it’s very expensive to allow AI to make assumptions that then get passed forward into your products and your business processes without that very careful check and watching the guardrails that you were talking about. The harnesses, the knowledge, right. One of the other things about token maxing is that people let the AI keep building on its own knowledge rather than making sure that you’re surrounding it with documentation and business knowledge and standard operating procedures. All of those things can actually be put into the processes, but they have to be done very deliberately.

Swapnil Bhartiya: The whole idea of an AI harness is getting a lot of attention these days. Can you talk about what does a harness actually do and why is it becoming a critical piece for enterprise AI deployments today?

Rob Hirschfeld: The funny thing about the harness is that in a lot of ways we’ve all been using harnesses without thinking about it. I think the transition here is using harnesses deliberately and making very intentional choices about it. And for a business leader, especially an executive, understanding this is absolutely critical because the harness is really the interface between the work that you have, whether it’s a chat or something more sophisticated, and the LLM, the AI model in the background. And so what you want to be able to do is separate the work that you need to be done from the model behind it. And the reason that’s essential is because that gives you the power to choose different models. That might mean switching to the appropriately sized model for the provider you have and being able to say, this work needs a complex model, this work needs a simple model and save cost. But it also allows you to then pull back and choose providers to use an open weights model, run a local model to be able to find different providers, set up your own AI infrastructure and then drive forward. And we’re helping companies who are building their own AI infrastructures actually deliver that capability. But it doesn’t do you any good if you haven’t been building the ability to switch into what you’re delivering and how you’re delivering that product. So the harness plays this absolutely essential role for you to be able to map the model into the work in a way that is flexible and scalable across your organization. So even if you don’t know when you’re going to be looking at additional AI models or additional AI sources, putting a harness in very deliberately and getting used to using the harness is an absolutely essential first step. And beyond that, the harness itself can do a lot of checks and guardrails and validation, filtering. What we’re finding is we embed the way RackN embeds AI into our product is that we actually embed a harness. And then the harness allows us to inject documentation, guardrails, information about how we want the AI to behave regardless of the model, behind the scenes, into the AI processing. So we don’t actually have to ship an AI LLM capability in the product. What we’re really doing is shipping a harness with the product. This is coming in future versions, but that thinking is absolutely essential. We are assuming that enterprises will want to control and manage the LLM and the processing in the background and still need the support of being able to connect their backing LLM AI infrastructure into whatever system you have and then feed in all that information. It sounds a little complex. It really is just a layer of software in your AI processing systems that you already have but aren’t using as deliberately as you need to.

Swapnil Bhartiya: Many enterprises are starting by putting GPUs in developer workstations and running local open weight models, which is a great way to get started. But why is that a reasonable place to begin but not a complete long term strategy?

Rob Hirschfeld: Well, there’s a couple of reasons why not, and one of them is innovation and pace of innovation. So I want to get there and put a pin on that and how you do it. But from an enterprise perspective, ultimately it’s going to be about control and auditability. So what we’re looking at here is if you’re running an enterprise system and you want to have your AI, your individual end users, you said developers, but it could be any end user leveraging a local model that is going to save you on processing capabilities. However, what you’re going to do is be limited to what that end user is doing on their machine when their machine is running. When you have access to it, it’s going to be very hard to audit and control and check. So the idea of having a cluster of AI resources actually gives you better cost controls, but it also gives you better auditability and governance to let you manage those pieces. And this is where the other dimension comes in. Inference does not necessarily require a GPU. There’s a lot of new chips coming into market that will provide AI inference capability and even running CPUs. And CPUs are getting new inference capabilities. So locking yourself into what’s on people’s desktops, it’s a good short term solution, but long term, having that dedicated AI capability with a heterogeneous mix of hardware so you can match models to capabilities and bring in new models whenever you need to, and then control to make sure that the models that you’ve approved are getting used. Those control points are absolutely essential and really from an enterprise perspective, critical to scale. And it’s worth noting when you look at how those pieces go, and that’s exactly what we’re helping companies spec and build today. The harness allows you to pull those pieces in. You guarantee an SLA, right? If you’ve been caught up in AI goes down, has a day where it’s not performing well and a lot of work gets stopped. You are going to want to be able to control even if you’re not on a frontier model. Having guaranteed access to models to run processes is absolutely essential because more routine processes can be run very effectively by open weights models. And along those lines, what you also need to think about is in that infrastructure and with those controls. And there’s one more critical point with this, which is that it’s not just your users, your developers and people doing this. Part of what you need to be thinking about building is agentic systems. And those agentic systems have to run somewhere. Also, you don’t want a room full of cracked lid laptops providing your agents. You actually want them running in controlled corporate infrastructure. And so all of these things work together to drive this realization that you want to have a dedicated AI infrastructure that can run your inference models in a controlled way, allow your agents to work especially offline or to provide batch operations in a controlled way. So all of these pieces fit together. So as much as I love running local models and empowering developers from an enterprise perspective, you need to be planning further out.

Swapnil Bhartiya: And now what is the path from that first step to owned AI infrastructure actually looks like for an enterprise?

Rob Hirschfeld: Wow, this is one of those ones. And actually it’s funny because VMware migration is similar to this in our books and experience is enterprises have a tendency to try and sequence out all these steps and they think they have to do this in a very orderly manner. Our experience actually is that the path should have a lot of parallel operations. You know immediately, right, that you can empower developers to do local models and look at harness. So harness is clearly a first step. But the people who are going to make an evaluation for your harness, your platform team, your developers, aren’t the same ones who actually are the ones who should be looking at how to run and spec AI gear and infrastructure. That’s mostly hardware teams and operation teams that need to understand how to set up and run AI gear, how to build Kubernetes from an AI workload perspective. And what I would encourage, especially because AI innovation is running so quickly, is that you need to be building these skills in multiple dimensions simultaneously. So you should be looking on how do I improve my delivery of AI infrastructure? How do I know what to buy, how do I know how to wire it together, how do I build these systems and get that skill set embedded in your organization? We see exactly the same thing going on with a lot of VMware choices, where people try to figure out what their exit out of VMware should look like and then work backwards. The reality is all answers require you to have better operational control of your infrastructure, more automation, better processes. This is what RackN does from a bare metal perspective. What we’ve seen is that companies who build the platform and the capabilities that allow them to then innovate on top of that platform end up moving a lot faster than the ones who figure out where they have to get to at the end of the trip. And don’t worry about any of the intermediate skills they’re going to need to build. Especially in today’s market. You need to be making sure you’re jumping through all the intermediate skills simultaneously to build this up. So don’t get tied up in, I need to figure out my harness, I have to figure out which LLM is best. You do need to do those things. But don’t wait on building AI infrastructure until after you’ve made those decisions. Start building all the skills across your teams so that they’re delivering the pieces that you need right out of the gate.

Swapnil Bhartiya: Now, one of the biggest challenges these days is getting AI gear. It is one of the hardest practical challenges right now. How are customers dealing with GPU and RAM scarcity?

Rob Hirschfeld: Yeah, this is something that’s affecting the industry as a whole because the cost of RAM and storage, in addition to the cost of GPUs and CPUs is going up to the extent where what we’re seeing people try to do is lock in server purchases as soon as possible so they can lock in prices and get shipments on RAM even today. And so this is one of those places where you need to realize you are not going to be able to have a preferred vendor, that if you’re used to buying from one vendor, you are not. They might not be able to supply you or their costs might get prohibitive for you to continue to stay on that platform of choice. What we see customers doing is they are being heterogeneous in design upfront. So they’re recognizing they’re either going to have to switch vendors and move, be able to intermix different vendors depending on their supply chain. They’re going to have to compromise on how they configure the systems and then have heterogeneous environments, even if they’re sticking with one vendor or they are sourcing RAM themselves or potentially planning to add RAM later, assuming it frees up in the market. And they are planning to keep systems in service for longer, which means patching and updating and revising them, or looking into the secondary market for these servers as the frontier CPUs and GPUs come off market. They’re very usable for inference and they’ll be available for you from these training labs. So you need to be looking very differently at the hardware infrastructure that you might have said, oh, I’m a Dell shop or a Cisco shop or an HP shop and been thinking that would save you. The reality today is that you’re going to get what you get. It even might be ARM servers with Nvidia chips to do this work and you need to be prepared for a higher degree of flexibility in what type of gear that you’re onboarding. And if you’re thinking that’s scary, I just want to never own hardware again. The challenge of this market is that renting your hardware or getting it from a service provider, you’re going to be paying that markup multiple times over. And so while it might be scary to pay more for hardware, owning the hardware and controlling the cost of that hardware is absolutely essential because all of these costs are being passed on with multipliers from the service providers. And that is a very serious concern. If you’re trying to manage your budget, it’s going to show up in your inferencing costs, it’s going to show up in your cloud and other component costs, it’s going to show up in basically any way you turn trying to avoid having hardware. The hardware costs are still going to get passed down to you.

Swapnil Bhartiya: Now let’s talk about inferencing. Why does inference infrastructure change the conversation around multi vendor sourcing and even previous generation hardware?

Rob Hirschfeld: And this is where things get a little tricky. We did spend a lot of time about two years ago very focused on what model builders needed, which is very fast interconnect. It focused on having a certain type of storage array, having these large clusters that were very tightly interconnected. Those are still important. And I do think that if you’re a large enterprise at some point you will look at doing small language models where you train a smaller model based on a larger model to become a subject matter expert on one topic because it makes the model smaller. And this, I think is what people need to understand about these models from an inference perspective is the way the inferencing system works is your interaction with the model. Everything you’ve done so far is a context window. The bottleneck in a lot of cases is how you manage and hold that context window. From a storage perspective, that takes a lot of RAM. It has to be loaded and fed into the model in a certain way. And as you build up a big system, especially a shared inferencing system, you actually have to load the context in between different users as they feed into those systems. And so that is actually one of the biggest performance bottlenecks on how an inference system is built. Now, the nice thing is that you don’t need to chain together tons and tons of systems with fiber optic networking between them to support training the model, but being able to build and maintain these systems, to inject the context windows into a model, store the model on the GPUs or in memory, all of those things are all still very demanding. And so there’s an interesting balance between the size of the models you use, how you load the context windows, how big those context windows can be, how effective the processing can be. Sometimes you can get bottlenecked not on the GPU, but on the CPU behind this. And so what we’re seeing is there’s a lot of ways to do inferencing. We’re still working out what the most effective model and system is. And in all likelihood, depending on the model and the workload that you’re doing, you’re going to have different systems for different models and different use cases. And that’s really important. When we look at the cost of these systems and how much RAM they have, and the CPU to GPU balance, or if they have special ASICs or special processors to run certain models, what you’re going to end up looking to do is optimize that mix. So the idea that you’re buying 64 machines, wiring them together as a VMware cluster, and then the differentiation is in the VMs, as we used to see in traditional enterprise IT. When we start looking at what a cluster would be for AI inference. Unless you’re super large scale and you’re running the same model and the same workload at incredibly large scale, it’s possible that you’re going to be having a much more heterogeneous mix of systems in your cluster, or you’re going to end up wasting a lot of capacity. And frankly, hardware is hard enough to get that you can’t afford to waste capacity. I don’t see that changing in the foreseeable future.

Swapnil Bhartiya: For executives who are trying to move past experimentation, what are the first infrastructure decisions they should make right now?

Rob Hirschfeld: I’m excited about this era that we’re entering because RackN’s mission goes back a generation in helping people be able to run hardware and bare metal themselves, own their own infrastructure and make that easy and possible and scalable. What we’re really doing is helping companies who are looking at all of this AI complexity and cost and the need for governance and control and service level agreements around their AI infrastructure and evaluate how do I own this core part of my business processing, how do I guarantee its availability, how do I make sure that I can afford the tokens that I do want to spend? Because most companies want their token capacity to grow exponentially over the next couple of years. What RackN does through our platform Digital Rebar is we make the bare metal layer specific, predictable and repeatable. We’ve embedded just amazing battle-proven operational practices into the platform. So when you bring any hardware, and I mean we literally have customers bringing up ARM servers alongside Intel AMD servers, whatever servers you plug into the system, we’re able to onboard, take through a repeatable process, get the platforms installed, join them into clusters, patch, update, network. All of those operational capabilities are baked into the platform. And it’s really important because in this world of incredibly heterogeneous, fast moving, expensive hardware, our customers are incredibly confident in their ability to onboard and use infrastructure in a way that we’re very proud of and I haven’t seen anywhere else in the industry. So a lot of people think about data centers and they think back to the 90s and it was incredibly bespoke and slow and expensive and very fragile and you were always patching and it was a real treadmill of operations. And they gave that up and moved to cloud. And now they’re finding that cloud is expensive, doesn’t have all the GPU resources they need, doesn’t have the controls and governance that they want, or they don’t want all that data flowing all over the place and they’re looking at how do I pull it back. What we’ve been able to do with the platform is make it so that they can just show up and start working right. We bring Kubernetes clusters up in a week or two weeks where our customers might have been spending months or even years trying to bring that hardware into conformance. And the beauty is once it’s built, they can just push a button, reset and bring it back and keep it in conformance. And as the systems come in and out, they’re automatically getting patched and updated, they’re getting the latest firmware patches, they’re getting on the networks. Some advanced networking is going on. We haven’t even talked about the DPUs and the SmartNICs and things like that. It’s a very complex environment out there. Our customers are able to just use their infrastructure and then focus on the workloads on top of it. That discipline, that out of the box capability and the ability to ingest whichever hardware they need to run whichever platforms they need on top of it is a game changer. And the confidence that you have that you can focus on getting your business done is really important. Because we’ve been talking about how challenging the environment is, how much things are changing, how much the expertise of getting this done is important. And I will promise you, if you ask an AI to help you with operations, they are not reliable sources. There’s no training data for the type of work that we do. And we watch customers try to ask AI to do operations for them and they end up with bricked systems, they end up with really poorly optimized systems, or they end up with just a huge mess. And so the ability to bypass that and just get straight into getting work done is absolutely essential. In this era, it’s very important to learn how to use the models and harnesses like we’ve discussed in detail. The thing that I would tell people is that be very careful with your experiments. What we have seen is that a lot of people buy expensive AI gear and then they set it up by hand and they end up with a bespoke environment that they don’t know how to recreate, they don’t know how to upgrade, they don’t know how to patch. You can’t afford to buy infrastructure and spend months and weeks fixing it, tuning it, doing all that stuff. Start with systems that have the automation. This is why we like to get involved with customers right in the start is that we can help you get your clusters up and running. But more than that, get them automated. So you can do a reset and a refresh and patch and control that actually lets you move these experiments faster. So don’t get confused that you have to turn every knob and build this from scratch. Like you might have the garage days or the basement ops themes where everybody had a server in their basement. That was a great way to learn last decade. Today you don’t have time for that. The systems are really complex and a lot of the things you learn in that lab phase won’t translate. I promise you they do not translate into production. So the sooner you get to production grade operations, the faster you are going to get to end results. And that’s the biggest mistake that we see people doing is they get stuck in perpetual POCs that don’t actually have any hope of making it into production. You have to start with production grade.

Swapnil Bhartiya: Rob, thank you so much for joining us and sharing these insights on what it really takes to move from AI experimentation to operational reality. I appreciate the conversation and I look forward to our next conversation. Thank you.

Rob Hirschfeld: I appreciate it. Thank you.

Swapnil Bhartiya: And for those watching, if you want to learn more, check out RackN and the work they are doing around Digital Rebar and enterprise AI infrastructure. I look forward to another great conversation. Thanks for watching.