Guest: Ben Lerner
Company: Espresso AI
Show Name: An Eye on AI
Topic: AI Infrastructure

CEO Ben Lerner explains how AI-driven workload optimization is reshaping the economics of modern data platforms — and why the Databricks vs Snowflake rivalry is good for everyone.

Over the past few years, Databricks and Snowflake have defined the modern cloud data stack. Both offer powerful platforms for analytics and AI, but as enterprises scale workloads, one thing has become painfully clear — data warehouse bills grow faster than expected.

Ben Lerner, CEO of Espresso AI, has made it his mission to change that. His team — composed largely of former Googlers — is bringing transformer-based intelligence to the problem of cloud data cost optimization. The company’s AI models analyze real query patterns inside Databricks SQL and automatically right-size compute, helping customers cut their bills by as much as 50–60% with no manual tuning required.

The Databricks Boom

Databricks has grown explosively in the past year, outpacing many competitors with its unified approach to data and AI. Lerner attributes that success to how Databricks allows organizations to work seamlessly across structured and unstructured data, making it ideal for modern AI workloads.

“Databricks gives data engineers and data scientists a shared space to collaborate,” Lerner says. “They can clean data, train models, and deploy analytics all on the same platform. That’s powerful.”

But that flexibility comes at a cost. As users scale compute clusters, especially in SQL workloads, the cloud bills can spiral. Databricks’ consumption-based model charges per compute unit, and teams often over-provision resources for performance assurance.

“Most organizations err on the side of safety,” Lerner explains. “They’ll allocate two or three times more compute than needed just to avoid job failures. Multiply that across hundreds of jobs, and you’re easily overspending by millions a year.”

Espresso AI’s Solution: Transformer-Based Optimization

Espresso AI was designed to solve this problem by applying AI to the very thing that creates the cost — query execution. The company’s optimizer uses transformer-based models to analyze query plans, runtime metrics, and historical performance data to predict the most efficient configuration for each workload.

“The same way large language models understand text context, our models understand workload context,” Lerner says. “They can see patterns that humans miss — for example, knowing when a particular query will benefit from a different cluster size or caching strategy — and then automatically adjust resources.”

This dynamic adjustment happens in real time. Espresso AI integrates with Databricks SQL APIs, so it can adjust compute allocation as workloads run, ensuring that clusters scale up only when needed and scale down immediately when idle.

“Our customers don’t have to change their code or pipelines,” Lerner notes. “They just flip a switch and the optimizer starts learning. Within days, they see measurable cost reductions without impacting performance.”

Why AI Optimization Beats Manual FinOps

Many companies already use FinOps tools to track cloud spending, but most of these tools are reactive — they show you where you overspent after the fact. Espresso AI aims to make optimization proactive and automatic.

“Traditional FinOps is about reporting; AI optimization is about prevention,” Lerner says. “Instead of telling you last month’s bill was 40% too high, our system ensures this month’s bill won’t be.”

For enterprises, this difference is crucial. Teams are stretched thin, and manual tuning is time-consuming. AI-driven optimization can operate continuously, learning from workload behavior and improving efficiency over time.

Databricks vs Snowflake: A Healthy Rivalry

Espresso AI originally focused on Snowflake, helping customers reduce their data warehouse costs by up to 70%. Lerner says expanding to Databricks was a natural step.

“Snowflake and Databricks represent two sides of the same coin,” he explains. “Snowflake is more traditional warehouse-centric, while Databricks is about unification — data lakes, ML, AI. Most large organizations actually use both, sometimes for different departments.”

That dual usage has fueled competition between the two companies, but Lerner believes the rivalry benefits the entire ecosystem.

“When two leaders push each other, innovation accelerates,” he says. “Snowflake has brought enterprise polish and governance; Databricks has driven openness and flexibility. The combination is good for customers.”

Espresso AI’s Role in the AI Infrastructure Layer

Beyond cost optimization, Espresso AI represents a broader trend — using AI to manage AI infrastructure. As generative and agentic AI workloads expand, the cost of compute is becoming the limiting factor. Optimizing these systems in real time will be critical.

“AI models are compute-hungry, and infrastructure efficiency determines whether your project scales or dies,” Lerner says. “That’s where we fit in. We’re bringing intelligence to the compute layer itself.”

He envisions a future where optimization systems are as standard as monitoring or observability tools. “In a few years, you won’t think about manual provisioning. AI will handle that automatically. The same way autoscaling changed cloud economics a decade ago, AI optimization will change it again.”

The Bigger Picture: From Visibility to Action

One of the biggest challenges enterprises face is turning visibility into action. Tools like Datadog and CloudHealth provide metrics, but decision-making still depends on human intervention. Espresso AI’s approach closes that loop.

“We’re bridging the gap between knowing and doing,” Lerner says. “Visibility is great, but if you’re not acting on it, you’re still wasting money. Our models act instantly — they don’t get tired, they don’t get distracted.”

That automation appeals to both technical and financial leaders. CTOs like that it optimizes performance without disruption. CFOs like that it delivers predictable cost savings.

“Our average customer sees payback within a month,” Lerner adds. “We’ve had teams call us after the first billing cycle to say they couldn’t believe how much it dropped.”

Looking Ahead

Espresso AI plans to expand beyond Databricks and Snowflake, bringing the same optimization approach to other compute-intensive systems like BigQuery, Redshift, and even vector databases that power agentic AI workloads.

“The principle is the same everywhere,” Lerner says. “Wherever compute is dynamic and expensive, AI can make it efficient.”

He also sees opportunity in helping teams use these insights for capacity planning and carbon reduction. “Optimizing compute isn’t just about cost — it’s about sustainability. When you cut unnecessary processing, you also cut emissions.”

Conclusion

As enterprises pour more investment into data and AI, cost efficiency has become as strategic as performance. Platforms like Databricks are essential for innovation, but without intelligent optimization, they can quickly drain budgets.

Espresso AI’s transformer-based optimizer offers a glimpse of what’s next — a future where AI doesn’t just analyze data, but also manages the infrastructure that runs it.

For Ben Lerner and his team, that’s not just a technical innovation. It’s a shift in how enterprises think about control, efficiency, and intelligence in the AI era.