AI Infrastructure

AI Token Costs: Governance Over Model Size | Rob Hirschfeld, RackN | TFiR

0

Enterprise AI token costs are climbing, and the instinct to swap foundation models for cheaper open-weight alternatives through an open router solves only the invoice line item. It transfers data sovereignty risk, obscures where processing actually occurs, and provides no audit trail for agentic operations. Self-hosting pushes the problem further: now the enterprise owns gear provisioning, model management, access controls, and the full operational burden that comes with them.

In this interview on TFiR, Rob Hirschfeld, CEO at RackN, breaks down why token economics is fundamentally an infrastructure governance problem, what model harnesses must do to make AI cost-efficient and auditable, and why chasing the largest parameter counts is already yielding diminishing returns for most enterprise workloads.

Guest: Rob Hirschfeld, CEO at RackN
Show: The Agentic Enterprise
LinkedIn: Rob Hirschfeld
Watch the discussion on YouTube

Here is what every platform engineer and enterprise AI infrastructure team needs to know.

Technical Deep Dive

Q: Why can’t enterprises solve AI token costs by simply switching to a cheaper open-weight model?

Rob Hirschfeld, CEO at RackN, explains that open-weight models do offer real cost advantages over foundation model APIs, but the trade-off is not just financial. Using an open router to substitute Anthropic or OpenAI with an open-weight model means the enterprise may not know where the actual processing is occurring, which creates data exposure risk on traffic sent across the wire. Governance controls, service-level agreements, availability guarantees, and repeatability all change when the model source changes, and most enterprises have not priced those risks into the decision.

“If you’re just trying to use an open router to swap Anthropic or OpenAI for an open-weight model, you might not know where the actual processing is being done, and that could compromise the data you’re sending across the wire.”

Rob Hirschfeld, CEO, RackN

Q: What operational burden does self-hosting an open-weight model place on an enterprise?

Pulling a model in-house to avoid the data exposure risk of an open router does not eliminate complexity; it concentrates it. Even renting GPU capacity from a neo-cloud, the enterprise becomes responsible for gear management, model behavior understanding, access governance, and the full operational stack underneath the inference layer. Hirschfeld frames this as an infrastructure ownership question: either someone else operates it and the enterprise accepts the associated risk, or the enterprise operates it and must build the capability to do so reliably.

“If you bring it back to run that model yourself, then all of a sudden, you have to worry about accessing and managing the hardware, doing the work, and building those systems.”

Rob Hirschfeld, CEO, RackN

Q: What is the real risk if enterprises do not govern their AI agents?

Hirschfeld identifies agentic AI as the point where ungoverned token generation becomes a security and business risk, not just a cost problem. Without governance, agents operating autonomously could collaborate in ways that cross compliance or competitive boundaries, including gathering competitive intelligence in ways the enterprise never intended. Controlling token generation is therefore not only about spend; it is about defining and enforcing the boundaries of what agents are permitted to do on the enterprise’s behalf.

“How do you make sure your agents aren’t going rogue and collaborating to attack your competitors to gather competitive intelligence? These are very real concerns, and you have to build safeguards against all of them into your systems.”

Rob Hirschfeld, CEO, RackN

Q: What is a model harness and why do enterprises need one before scaling AI?

A model harness is the governance and orchestration layer that sits between enterprise workloads and AI models. Hirschfeld points to tools like Claude Desktop as consumer-facing examples of a harness: they let users choose and substitute models at runtime. At enterprise scale, a harness must also enforce guardrails, provide audit trails, incorporate skills and standards, secure agentic operations, and make model substitution operationally safe. Enterprises accelerating AI adoption need to ask whether their current tooling provides all of these capabilities before committing to a model strategy.

“Companies that are leaning heavily into AI and trying to accelerate its adoption need to be asking: How are you creating harnesses? How do those harnesses allow you to switch models, provide security and guardrails, learn, incorporate skills, establish standards, and enable auditing within your systems?”

Rob Hirschfeld, CEO, RackN

Q: Are the biggest, highest-parameter AI models still the best choice for enterprise workloads?

Hirschfeld argues that improving frontier models is already showing diminishing returns for most enterprise use cases. Two-generation-old models already deliver strong analysis, thinking, and planning outputs for scoped tasks. The determining factor for output quality at the enterprise level is not parameter count but how well the model is governed, how its processes are managed, and how well agents are coordinated. Chasing the largest model available delays the harder and more valuable work of building the governance layer.

“You can get amazing thinking, analysis, and planning from models that are two generations old. It really comes down to how you’re governing, controlling, and managing their processes and helping them coordinate their work.”

Rob Hirschfeld, CEO, RackN

Q: When does a smaller, focused AI model outperform a larger general-purpose model for enterprise tasks?

Smaller models outperform larger ones on scoped, well-defined tasks where the cost per token matters and where the governance layer can route tasks to the appropriate model. Hirschfeld’s point is not that smaller models are universally better, but that enterprises fixated on the largest available model are skipping the architectural decision that actually drives performance and cost efficiency: building a harness that matches the right model to the right task, enforces standards, and substitutes models safely as the landscape changes.

“Smaller models make a lot of sense. It really comes down to how you put your governance in place. Companies should make sure they’re not just chasing the fastest, biggest model but are actually thinking through what their governance and control processes should look like.”

Rob Hirschfeld, CEO, RackN

Resources & Documentation

  • RackN, infrastructure automation and AI governance platform for enterprise operations

***

👇 Click to Read Full Raw Transcript

Swapnil Bhartiya: Now let’s talk about token economics because it is a huge concern for every enterprise right now. Fine tuning or quantizing models can cut cost, but it can also hurt quality. Where do you see this cost versus quality trade off heading?

Rob Hirschfeld: Token economics is a really major concern. And as a rule, looking at open weight models as a solution to token economics can save you a lot of money rather than going to the foundation. However, there are a lot of things to consider in that besides just the cost, governance controls, service level, availability, repeatability. Right. So one of the things that’s really important here is if you’re getting access to an open weights model, one of the benefits might be that you have better governance and controls around that model. But if you’re just trying to swap, like use an open router to swap Anthropic or OpenAI for an open weight model, you might not know where that actual processing is being done and it might compromise the data that you’re sending across the wire. If you pull it back to run that model yourself, then all of a sudden you have to worry about getting access to the gear and managing the gear and doing that work and building those systems up. And if nothing else, you’re going to be even if you go to a neo cloud and get the gear from a rental perspective, you’re now responsible for the management of it. And you have to understand the model behaviors, you have to understand the model you’re using, you have to understand how you’re governing access and control. So I think token economics is a very critical thing and companies need to be thinking about how do they ultimately control their AI destiny, their token generation destinies? Because everybody should be thinking about this. How do you make tokens inexpensive enough that you can let agents go crazy doing processing actions and really working on your behalf? And how do you have the governance and controls? How do you make sure that your agents aren’t going rogue and collaborating to attack your competitor to get competitive intelligence? Right. These are actually very real concerns and you have to build all of that into your systems and your plans for these systems as you go forward. So I think these are not easy questions. I don’t have all the answers for them, but I do know that at the bottom of all this is going to be how well you can operate and manage infrastructure. Because either you’re going to let somebody else do it or you’re going to be running it yourself and managing that infrastructure. And I think very few enterprises are ultimately in the position to just relax those rules and not worry about it. I think you have to ultimately be able to operate.

Swapnil Bhartiya: Now, shifting from hardware to models, there is an obsession with the biggest, most parameter heavy model is the best model. But that is not true. Bigger is not always better because for a lot of tasks a smaller model, very focused model can get that job done in a much better way at a cheaper cost. What’s your advice to enterprises who are fixated on chasing the biggest model instead of right size one?

Rob Hirschfeld: There are and companies need to be building their practice of using a harness right. And if you’re using like cloud desktop, that is a harness. It lets you choose models as you go. But we are seeing more and more generic harnesses or general purpose harnesses. And that’s one of the things that lets you choose and substitute models that also helps create governance and controls around your agentic operations and your agents. And so one of the things I would say is that companies that are leaning heavily and trying to accelerate their use of AI, which should be everybody listening to this, you need to be asking how are you creating harnesses? How are those harnesses letting you change models? How are those harnesses securing, providing guardrails, learning, being able to incorporate skills and create standards and also audit in their system. So we’re entering a phase where I think improving models is starting to have diminishing returns. There’s certainly some amazing things going on with models, but I can speak for RackN and I’ve heard this more generally. You can get amazing thinking, analysis and planning from two generations old models and it really comes back to how you’re governing them, controlling them, managing their processes, helping them coordinate their work. So you’re entirely right. Smaller models make a lot of sense. It really is how you put your governance in place. And that’s, you know, companies should be making sure they’re not trying to just chase the fastest biggest model, but actually think through what their governance and control processes look like.

Secure MCP Migration with Edge Protection | Du’An Lightfoot, Akamai | TFiR

Previous article