Guest: Graham Neubig
Company: OpenHands
Show: The Agentic Enterprise
Topic: Agentic AI

If you’re deploying AI agents for software engineering, you’re facing a critical decision: which large language model actually delivers when it comes to real-world coding tasks? It’s not about synthetic benchmarks that test basic capabilities. It’s about actual issue resolution, front-end development, and production workflows. The answer is far more nuanced than most organizations realize, some models are fast but expensive, others are cheap but inconsistent. That’s exactly the problem the OpenHands Index was designed to solve.

Graham Neubig, Chief Scientist at OpenHands, leads the effort behind the OpenHands Index, a continuously updated leaderboard that evaluates large language models across a broad variety of software engineering tasks. Unlike narrow benchmarks, this index assesses models from three critical perspectives: accuracy, cost, and time to resolution.

“One of the features of OpenHands is that it’s model-agnostic, so we can use any models,” Neubig explains. “Every time a new model comes out, our users ask us, ‘Is this a good model? Should we try it out?’ The OpenHands Index is a benchmarking effort we created to answer this question very quickly.”

The index goes beyond the well-known SWE-bench, which checks whether agents can solve issues on Python repositories. OpenHands evaluates five different benchmarks covering front-end development, software testing, information gathering, and more, providing a comprehensive view of real-world software engineering capabilities.

The Current Winners and Why They Matter

Right now, according to the latest OpenHands Index results, Claude Opus 4.6 from Anthropic sits at the top. “It’s at the top both from the point of view of accuracy and speed to resolution,” says Neubig. “It finishes issues very quickly. The only problem is it’s very expensive, it’s one of the most expensive models.”

That expense factor is critical. For organizations deploying coding agents at scale, API costs can quickly balloon into millions of dollars. That’s where the cost-versus-capability trade-off becomes essential.

“For cost optimization, our favorite right now is MiniMax, which is an open-weights model that was just released very recently,” Neubig notes. “It’s the first model we’ve seen that is kind of comparable with Claude Sonnet, but it’s about one-tenth of the price.”

The OpenHands Index visualizes this through a Pareto curve showing cost versus accuracy. If you need maximum capability, you choose one model. If you need to optimize for cost, you choose another. The spectrum between those extremes gives teams the flexibility to make informed decisions.

Task-Specific Model Selection

What makes the OpenHands Index particularly valuable is its recognition that different tasks benefit from different models. For pure coding tasks, Claude Opus performs exceptionally well. But when building an app entirely from scratch, Codex can be better because it carefully follows instructions and continues working until completion.

“Codex tends to very carefully follow instructions and continue working until it’s really done, whereas the Claude models can sometimes stop halfway through or not fully finish all of your instructions,” Neubig explains.

For reusable workflows like pull request reviews or library upgrades, tasks that run repeatedly, expensive flagship models aren’t necessary. Less expensive open-source models can handle these routine operations effectively, dramatically reducing operational costs.

The Deployment Reality

In practice, most organizations today focus on a single language model, largely because they lack clear guidance on when to use which model. “It actually requires quite a bit of expertise to be able to really manage that,” Neubig says. “Most organizations we talk to, especially larger enterprises, aren’t quite familiar enough yet to do that.”

But that’s changing. As deployments mature and API costs mount, organizations are increasingly interested in strategic model selection. “The cheaper models are getting so good nowadays that for a lot of the simpler tasks we do, we really don’t need a really expensive model,” Neubig observes.

The decision-making process also varies by organization type. Large enterprises in security-sensitive industries often have a trusted language model, sometimes deployed on their own infrastructure, and stick with it. In contrast, open-source community members immediately test every new model release, looking for optimal performance.

Beyond Benchmarks: Verification and Observability

The OpenHands Index provides a crucial first pass for model selection, but Neubig emphasizes that deployment success requires more. “The OpenHands Index is just meant to be a first pass,” he says. “In the end, it’s: is this working in deployment? Is it meeting our cost and other requirements?”

OpenHands is building observability capabilities in partnership with Laminar, allowing organizations to gather agent conversations, analyze them with language models, and aggregate insights. This enables teams to improve prompts and create “skills”, instructions on how to use an agent for particular use cases.

The next frontier is verification. “Code is cheap now, it never was before,” Neubig says. “You can generate as much code as you want, but good code is not cheap because you need to check that it actually works, actually meets requirements, and doesn’t add a lot of tech debt.”

OpenHands is training models to predict whether generated code will survive into the future, combining traditional static analysis and unit tests with AI-powered code review. This verification layer will eventually merge with the OpenHands Index, creating a comprehensive framework for deploying and managing coding agents in production.

For engineering teams navigating the rapidly evolving landscape of AI coding agents, the OpenHands Index provides much-needed clarity on which models deliver for real-world software engineering tasks, and at what cost.