Why Existing Server Tools Can’t Manage New AI Hardware | Rob Hirschfeld, RackN | TFiR

Swapnil Bhartiya: Every enterprise leader is nervous about the same thing right now. Where is our AI budget actually going? Token costs, as we know, are climbing. A new wave of AI hardware is hitting the market and it’s getting harder to separate real innovation from marketing hype. And to cut through all of this, we have with us once again Robert Schulz, CEO and co founder of Reckoning. Now, before we get into hardware, every product today wears an AI enabled label. How much of that is real enterprise adoption versus just AI washing? What percentage of enterprises are actually using AI in meaningful production day today?

Rob Hirschfeld: Oh my goodness. I mean, AI adoption in the enterprise, as far as we can tell, is across the board. Probably people are overshooting the mark a little bit and turning themselves into reverse centaurs, meaning they have a lot of people who are using AI instead of thinking. And we’re in a correction curve where companies are learning how to use AI to take over and assist people. But there’s starting to be a lot more sensitivity over people who are just acting as a go between from an AI output and they just send you an AI output without much filter and thought. So we’re finally starting to see some rebound from, from that perspective, AI graphics aren’t as welcomed in slide presentations as they used to be. So we’re seeing a tremendous amount of adoption. And now we’re starting to see some people separate AI thinking from human thinking and understand what AI does really well along those lines, though we’re still at an early point where a lot of those are human interactions, human to AI interactions, the real agentic workloads, where agents are processing and doing work in the background as digital workers is still a little bit off, where the people who start to enable that quickly learn that the AI as a digital worker has a tendency to need a fair bit of supervision. And so you have to factor that in putting AIs into individual apps or workflows to do analysis or things like that, that has been showing up quite strongly in systems and we’re seeing a bit of overshoot there too, because we’ve done a really good job of everybody learning how to read mannered speech, that very specific AI dialect, and people are starting to push back against it. So as powerful as this is, using it correctly and keeping humans in the loop is absolutely essential.

Swapnil Bhartiya: Now let’s talk about this new generation of integrated AI server chassis. Hardware is hard, but one spec changes and suddenly your whole stack doesn’t fit. So talk about what’s really different about these new systems and what should enterprises watch for to separate real innovation from the hype.

Rob Hirschfeld: So there is a tremendous amount of innovation going on in the hardware space. The AI use cases, the AI leaders are really driving new new chassis designs, new components, new capabilities, new performance requirements, and then all of the subsequent drivers configuration changes, capabilities on top of that. So when we look at servers today, servers have been relatively stable for the last decade or so, with the biggest innovation being Redfish coming in to provide some degree of standardization, but still very OEM specific. The generation of tooling that we’re seeing now with SmartNics or GPUs, with specialized storage and tuned shared storage and memory, distributed buses, definitely GPUs, and now more and more power management, all of those capabilities are driving new components in the systems, new configuration tooling. That tooling is not reached by the OEM tools that you might be used to using. And so what we’re finding is people who are really using AI tools heavily have to be able to manage the whole box and all of the components in the box, and they’re not going to get the lift that they might be used to from any of their previous generation tools. So you can, if you’re a Dell shop, say you can’t count on Dell to do all that type of management and automation. And what we’re finding is because of supply chain and availability, it’s much harder to stay brand loyal and get the benefits of trying to have brand loyalty. That’s going to create situations where you might not be able to buy the servers you want, or with the RAM or the configuration you need. And so all of these forces have come together to make the landscape of bare metal infrastructure significantly more challenging to even do the basics well, let alone really operate at scale. Like if you’re spending $200,000, $250,000, and that’s probably the low end for some of these AI systems on a server, then you can’t afford to let it sit on the loading dock. You can’t afford to let it sit while you’re configuring it. You can’t afford to let it sit while you’re patching and updating it. Those activities with that price of server, it has to get in the field, it has to get in use as soon as it possibly can. And we’re not even talking about the delays in acquiring the gears. It’s getting harder and harder to actually get the infrastructure to show up on your loading dock. So the time pressure and time scale on these systems is just remarkably compressed.

Swapnil Bhartiya: On the hardware side. Training and inferencing have traditionally run on the same class of chips, but we are now seeing purpose built inference silicon like OpenAI’s work with Broadcom. Do you see training and inferencing? The hardware should fully diverge or will GPUs from Nvidia, AMD and other keep dominating both workloads?

Rob Hirschfeld: One of the things that’s happened this year is that the split between hardware for training and the hardware for inferencing, inferencing has taken over as the dominant workload and we expect that to continue to accelerate. The challenge that people need to reflect on is we will have a generation of inference specific chips. That’s not going to replace CPUs, it’s not going to replace RAM, it’s not going to replace storage. It might impact GPUs and lessen a little bit of the demand on GPUs. But fundamentally models are getting larger for companies that want to do real work. Even with open models, you’re talking about very powerful systems with a lot of RAM running GPUs or some type of inferencing chip. And so the components and capabilities around that infrastructure is pretty constant. So I wouldn’t look for anybody to say, you know what, we’re going to have an alternative to Nvidia GPUs. That really isn’t going to change very much about what you have to deal with. It’s not going to change the heat, power, performance, all of those requirements are all still bundled together.

Swapnil Bhartiya: Now let’s talk about token economics because it is a huge concern for every enterprise right now. Fine tuning or quantizing models can cut cost, but it can also hurt quality. Where do you see this cost versus quality trade off heading?

Rob Hirschfeld: Token economics is a really major concern. And as a rule, looking at open weight models as a solution to token economics can save you a lot of money rather than going to the foundation labs. However, there are a lot of things to consider in that besides just the cost, governance controls, service level availability, repeatability, right? So one of the things that’s really important here is if you’re getting access to an open weights model, one of the benefits might be that you have better governance and controls around that model. But if you’re just trying to swap, like use an open router to Swap anthropic or OpenAI for an open weight model one, you might not know where that actual processing is being done and it might compromise the data that you’re sending across the wire. If you pull it back to run that model yourself, then all of a sudden you have to worry about getting access to the gear and managing the gear and doing that work and building those systems up. And if nothing else, you’re going to be, you know, even if you go to a NEO cloud and get the gear from a rental perspective, you’re now responsible for the management of it. And you have to understand the model behaviors, you have to understand which models you’re using, you have to understand how you’re governing access and control. So I think token economics is a very critical thing. And companies need to be thinking about how do they ultimately control their AI destiny, their token generation destinies? Because everybody should be thinking about this. How do you make tokens inexpensive enough that you can let agents go crazy doing processing actions and really working on your behalf? And how do you have the governance and controls? How do you make sure that your agents aren’t going rogue and collaborating to attack your competitor to get in competitive intelligence? Right. These are actually very real concerns and you have to build all of that into your systems and your plans for these systems as you go forward. So I think these are not easy questions. I don’t have all the answers for them. But I do know that at the bottom of all this is going to be how well you can operate and manage infrastructure. Because either you’re going to let somebody else do it or you’re going to be running it yourself and managing that infrastructure. And I think very few enterprises are ultimately in the position to just relax those rules and not worry about it. I think you have to ultimately be able to operate.

Swapnil Bhartiya: Now, shifting from hardware to models, there is an obsession with the biggest, most parameter heavy model is the best model. But that is not true. Bigger is not always better because for a lot of tasks a smaller model, very focused model can get that job done in a much better way at a cheaper cost. What’s your advice to enterprises who are fixated on chasing the biggest model instead of right size one?

Rob Hirschfeld: They are. And companies need to be building their practice of using a harness. Right? And if you’re using like cloud desktop, that is a harness, it lets you choose models as you go. But, but we are seeing more and more generic harnesses or general purpose harnesses. And that’s one of the things that lets you choose and substitute models that also helps create governance and controls around your agentic operations and your agents. And so one of the things I would say is that companies that are leaning heavily and trying to accelerate their use of AI, which should be everybody listening to this, you need to be asking how are you creating harnesses? How are those harnesses letting you change models? How are those harnesses securing, providing guardrails, learning, being able to incorporate skills and create standards and also audit in their system. So we’re entering a phase where I think improving models is starting to have diminishing returns. There’s certainly some amazing things going on with models but I can speak for RackN and I’ve heard this more generally. You can get amazing thinking, analysis and planning from two generations old models and it really comes back to how you’re governing them, controlling them, managing their processes, helping them coordinate their work. So you’re entirely right. Smaller models make a lot of sense. It really is how you put your governance in place and that’s, you know, companies should be making sure they’re not trying to just chase the fastest biggest model, but actually think through what their governance and control processes look like.

Swapnil Bhartiya: RackN is a heavy user of AI internally, real dog fooding is going on there. What lessons have you learned that other enterprise leaders, your customers can learn from your own lessons?

Rob Hirschfeld: There’s a lot of things that RackN is doing from a software perspective. You know, we write software that helps people manage data centers and we leverage AI within our knowledge base and under very careful governance to accelerate how we develop and manage and improve security, improve completeness, you know, test edge cases, test our product. And we’ve seen a tremendous improvement in development velocity and also what each engineer can take up and perform and get done. And we’ve been able, we’re coming out with an interim release that includes agentic capabilities wired deeply into digital rebar that will help operators do more and use the product more efficiently. It doesn’t replace the operational knowledge and understanding of what the systems need to do. And so I would tell people who are following us in the journey that ultimately if you’re having an agent write software, it’s still a software engineering process. You need to have the gates, the controls, you need to have the subject matter experts. You need to constrain the actions that you’re asking for. What we saw is that when agents are given free reign, they don’t write good code, they take risks, they break things that shouldn’t have been changed at all. There are really significant risks. And in our space where you’re dealing with production, operational infrastructure, letting an AI say oh that was a good point, I should have thought not to reboot the server in the middle of a job run, those guardrails and controls are absolutely essential and they have to be part of your process. So it’s not just can you use AI to write code or reason or plan better, it actually comes back to what processes you’re following to coordinate, synchronize that work to allow AIs to go do a lot of work in places where it’s safe to do work and be heavily supervised in places where it’s riskier to do work and then create the tests, which is really just software engineering accelerated. Our experience has been as a product company that software engineering has always been a strength of what we deliver into an operations environment. And a lot of RackN’s advantage is not the fact that we’ve written software that does operations tasks. It’s that we’ve written software as software and we treat it as a software development life cycle with releases and immutable code and customer to customer standardization. Those are software values that we’ve embedded. And I think you run the risk of having an AI run around like a jackrabbit and just say, yes, of course, I’ll do that, I’ll do that, I’ll do that very eagerly, trying to please you to do things. And that’s just going to accelerate. If you don’t have strong processes and controls, it’s actually going to accelerate the result of that. And most organizations are already suffering from people doing whatever they want, whenever they want, without a lot of controls and oversight. So AI is just going to make that faster.

Swapnil Bhartiya: With all this excitement around AI, cost is becoming something which is impossible to ignore. And so is the marketing hype. Are we finally seeing real pushback against flashy AI marketing? All the fluff and against the idea that AI is basically all about one company’s hardware stack.

Rob Hirschfeld: There’s definitely AI exhaustion. We, you know, the challenge is we’re all using this and so we know that if we’re reading something where or watching something that AI did, the reasoning for it and there doesn’t look like there’s a human touch on it. We know that we could just ask our own AI to do that same work. And so we definitely see people wanting that human touch. If somebody’s working with RackN and they’re trying to learn how to operate their data center better and using our process, our product, they don’t want to be talking to an AI chatbot when they need help. We’re assuming they already have tried that. They have already asked for support from AI. Right. We’re embedding an agent into the product. They’re going to go through that agent by the time they’re talking to us, they’re ready to talk to a person. And I think that human touch and the desire to interact with other people is going to be, it already is very high. I’m about to give a presentation that has no AI graphics in it, no AI tells in it, because I wrote it myself, I put time in and I gave thought and I’m going to be doing it in front of an audience of people. And you know, if I’m asking for their attention, I’m going to give them my time. I’m not going to just produce something that they could have produced themselves with a couple of prompts. So I do think there’s a very real pushback now as people are looking to figure out what an authentic human interaction is and also what is a lift beyond what their product was giving them or what they can do themselves. That’s a question that I think we, RackN as a product company have to answer in a very compelling way. Right. Why don’t you just prompt to build whatever product you want? We’ve seen this across the board in SaaS companies and the reality is we one, we’re using AI also, so we’re faster in our domain area than anybody else is going to be. And we are already covering the ground and the mistakes and the changes and the fixes and the reactions and providing the discipline. And what we see over and over again is if you just ask AI to solve something to the best of your prompting capability, the AI is going to answer to your vision of what that capability is. And if you don’t have a lot of domain expertise in anything you’re doing or in the thing that you’re asking AI to build for you, then you’re at risk of how deeply the AI goes to satisfy your request. When it comes to data center operations, no team on the planet has deeper knowledge than the team at RackN. And our ability to express that into a product and then provide an API that’s repeatable and brings forth all of the automation and process is a critical differentiator in what we’re going to do. And I think when people look at using AI, they need to understand where their domain expertise and what their knowledge is. And even though the AI sounds like an expert, a lot of times it doesn’t truly have expertise to understand the consequences of what it’s building. And so you have to be very careful letting AI speak as an authority in areas that you don’t understand, whether it’s accurate authority or not. I find that it makes mistakes about markets, about technology, about safety, all the time. And you have to be aware.

Swapnil Bhartiya: There is a narrative that traditional software development is dying, that AI writes the code and humans just review it. In my opinion, it feels almost democratizing the way Photoshop changed photography. What’s your take? Is AI really killing software development as we know it, or is it really augmenting it and making it better?

Rob Hirschfeld: Yeah, where there’s a very tight loop that people can engage with, like a photograph or writing, those are places where that tight loop makes things easier. But if you’re dealing with what we do, which is I need to configure a server, there’s no record of how to configure a server in the general knowledge base those AIs were trained in. So they have to reason and rationalize what they’re doing. There are certain things you don’t have time to interact with a human system. There’s inertia. Understand what it did, look at its reaction. Try to learn from that. You have to understand where the learning is occurring in that loop and how much time it takes to learn and what it takes to build those cycles up. And so when you start dealing with physical systems, when you deal with things that have high expertise, components that are reliant on things being right, not just human judgment or style choices, then you really need to reevaluate how quickly AI can perform that work for you. We’ve all had this experience when having AI build a graphic or do photographs and you say, okay, AI, this looks great. I just need you to make the circle red instead of green. And then all of a sudden it’s producing hexagons and you’re trying to back it up to say, no, I didn’t ask for a hexagon. Go back to a circle, I want a red circle. And the whole context is lost. And that graphic has basically fallen apart. We have that experience intimately with AI. I love to use AI as a writing assistant and a peer in writing processes also. And I get to a point where I’ve been writing something and AI’s arguing with me. And it might latch onto a point that doesn’t make sense, or it might inject a bias into its writing that then creeps into everything it does from that point forward. And you have to be incredibly careful about what’s going on. So as these tools get more and more intelligent, then they also start to inject their own biases. They start to have confidence in assumptions that they’ve made that may not be correct. So I agree with you. It’s a fascinating age that we’re in the middle of and we have to understand that these systems are not as direct to control. It’s very, you know, through two layers of glass in these systems. And if you want reliable, repeatable systems, you’re going to have to break that glass and have a locked system where you just get reliable pieces. I don’t want, you know, I do allow AI to drive for me, but those are incredibly constrained systems tested exhaustively with a whole bunch of safeguards. I would say they’re hard, they’re not chat GPT deciding how to drive. It’s automation. And that’s actually very different than AI.

Swapnil Bhartiya: I come from an open source background, so I always trust open source code bases. The reason is that it doesn’t tie its success or vision to that company’s bottom line. The success of open source is closely tied to the success of its users. So the equation changes. It’s not driven by the company that manages, but users who are using it. So when it comes to AI and your cynicism, your lack of trust for AI, is it a distrust in AI in general or the current stack of AI?

Rob Hirschfeld: No, this is a very good point. So this is, I think, a subtle difference between LLM as a reasoning engine and automation. So my comment was there’s a difference between AI and automation. Automation, which is what most software has historically been, means that you have a deterministic algorithm that can be tested, produce a repeatable result and used over and over again. And so when we’re writing software and our product has a huge amount of automation and support around automation, we can take a type of server with a configuration and an operating system and a platform and drive that result over and over and over again. Highly repeatable, highly tuned. That is automation. AI, and the way we talk about AI today, what’s happening is it’s reasoning out every action that it takes. And I wouldn’t want a person to be sitting down at a server and trying to figure out how to configure a server over and over again. We don’t even like it when humans have a list of instructions that they’re following because they’re not particularly reliable following a list of instructions, they get bored and move on. The challenge we have with using AI in its current version to do that type of work is that it makes decisions dynamically as it’s going. It’s predictive. And those predictions aren’t necessarily repeatable. They’re subject to a lot of inputs, a lot of conflicting rules. They are not a repeatable set of operations, and they don’t perform the same way from the same prompts. We all have this experience, they’re not designed to do that. And if we want to run things and have predictable outcomes like driving a car, then that is an automation. We don’t want an LLM interpreting how to read a stop sign new every time it encounters a stop sign. That would be algorithmic. Now you might build systems together so that 99% of the driving is automation, and then you add an LLM on top of that. But that’s a different model. It’s exactly what we’ve done from an AI perspective for RackN with Digital Rebar, an incredibly deterministic system that has built in processes that operate in a reliable way. And then we can add AI on top of it to help a user accomplish a goal based on that toolkit of things it knows how to do. Incredibly productive use of AI, or interpret a result, or read a log and tell you what’s going on. But to invent a new way to do something dynamically when it sees it for the first time, which is what an LLM approach would be, you know, it would be like asking AI to generate photos of you over and over and over again. Every time you needed a picture, generate a new photo. And as we see those things, they have some slippage. That’s why AI has so many tells. That’s why AI video looks a little weird. That’s why AI pictures look a little weird. I assume things will progress, but we want to have deterministic outcomes that you can build on top of, build intelligence on top of. It’s a long answer, but to me it’s an important difference for how people need to think.

Swapnil Bhartiya: Looking ahead, beyond the hype, beyond AGI talk, what’s the one thing about enterprise AI that you think will genuinely surprise people?

Rob Hirschfeld: Yeah, I think there’s a real challenge here because a lot of the enterprises we’re talking to are still trying to deal with how do I replace VMware? It’s a real question for them because they’re spending money on that. They have to fix their estate, but they’re grinding through it. We talk to companies that act like they have a lot of time to invent their own operations, processes and figure things out on their own and where they think AI is going to suddenly show up and help them iterate faster through things, where they’re already struggling with process and process controls. I think that when enterprises look at the current AI landscape, they need to not just be thinking about how do I use AI, but they need to be thinking about how they make decisions, what they’re spending time operating on, where they’ve been getting stuck before and rethinking. This is a theme in our conversation. They need to be rethinking how they’re making decisions because they’re getting stuck on things. But we’ve always done it this way. We love this process. It’s really deeply embedded. We have to do it this way. And they’re missing the idea that they don’t have the time to reset and fix all their processes. They need to be moving, bringing their general operational knowledge up much more aggressively than they’re doing. And so I think that this isn’t just how do I use AI more effectively here, it’s how do I recognize that AI is going to be driving me to have to change my processes or improve my processes, move on, move faster on proven things and you might spend more money on other systems. Right? You might keep VMware in place even though you don’t like them as a vendor or it’s expensive because you don’t want to take the time to do that replacement. People need to make those decisions and move on. And where we started to bring this full circle is you can’t just assume because something worked in the past, it’s going to keep working in the future. We are seeing across the board from an operations perspective on people who manage their own infrastructure, and I think you’re going to see this is true in cloud also, that if it’s working today, the next generation stuff showing up is not going to work the same way. And you’re going to have to be prepared, you’re going to have to find partners who are already doing that work so that you can smooth over those bumps because you don’t have time in terms of your organization to absorb all those shocks.

Swapnil Bhartiya: Rob, thank you so much for this candid conversation on what really matters for enterprise today and for cutting through all that hype and myth. Once again, it was a great conversation. Those who are watching, please go and check out RackN’s blog and videos because they are doing a lot of work in this space that can help your organization as well. And once again, thank you so much for chatting with me and I look forward to chatting with you again. Thank you.

Rob Hirschfeld: My pleasure. Thank you.