Why Enterprise AI Pilots Fail in Production | Leo Brunnick, Cloudera | TFiR

Swapnil Bhartiya: Today, enterprise data lives almost everywhere on prem, in multiple clouds and at the edge behind firewalls that it cannot leave. So when it comes to AI, AI has to go to the data and what it costs to run there decides whether a project survives or fails. And of course cost becomes a big factor. Cloudera’s research says that 88% of enterprise AI projects stall before the production. We have seen this pilot to project story all the time, but the blocker is in the model is this gap between data and AI and that’s the gap Cloudera is trying to fill while its Nvidia partnership also try to tackle cost and performance. And today we have with us Leo Brunick, Chief Product officer at Cloudera. To get into all of that. First of all, Leo, it’s great to have you on the show, great to

Leo Brunnick: be here, thank you.

Swapnil Bhartiya: Let’s set the stage for our audience. Of course, almost everybody knows Cloudera, but just to put into context so they understand who they’re listening to and why it matters, just give a quick background on Cloudera, please.

Leo Brunnick: So Cloudera, which was formed by the combination of Horton Works in Cloudera some six years ago or so, hortonworks and Cloudera were the unicorns of the big data era. If you go back between 15 and 10 years ago and you know, it was, you know, I think of those early days as what I called the era of control, right? Getting control of that data for the first time. But you had, because it was primarily an on prem game, it was an easier game to control, cost to control, security to have a more predictable amortization based way of doing your work. And that changed seven, eight years ago was the big pop when the hyperscalers took off and some other players that were built for the cloud, built for the hyperscalers, the winners in that being databricks and Snowflake, they took off because that’s the era that I call the era of convenience. Right? It was just easy. You couldn’t do as many custom things, you had to put the data out there with it. But gosh, it was so easy to get projects done. You could run a credit card and be doing something interesting in an hour and not worrying about waiting for it or doing anything else. And then if you go back, gosh, two years ago, even, even two years ago, everybody thought, you know, at the, at the end of the day, everything is going to end up in the public cloud. That’s just the way it’s going to be. And you might be on year eight of your three, three year journey to aws. But you were going to get there someday. And over the last couple years, for a series of reasons, the market, nothing changed. Just that the, the realization that that hybrid was the actual end state. It wasn’t a way station on the way to being somewhere else. It was the end state and it was the desired end state. Right. You know, we work with mostly Global 2000 customers, big customers, you know, doing all of their data engineering, their data warehousing and more and more their AI projects. And for those companies, their crown jewels, their, their alpha is their unique data. And you know, and if you go over the last couple of years, the concept not of just AI being private and being local, but even more now you see just the overall concept of sovereignty, right? So sovereign nation, sovereign state, sovereign whatever it is, the, the sovereignty layer has come on top. So you, you get to today and it’s just general common sense. You don’t move data to AI, you move AI to data. And as you said, that’s on prem. That might even be an air gapped environment. It’s going to be on public cloud, maybe multiple public clouds. It could be on a sovereign cloud or a NEO cloud and it’s going to be out at the edge. And so, you know, Cloudera, so this, this new era that’s come in over the last couple years, I call the era of convergence. All that control and all that convenience have to come together. So you know, I joined Cloudera, you know, almost two years ago, about 20 months ago, brought with me a world beating CTO, Sergio Gago. And we, we have been building what we call the Data platform of 2030 and it’s called Anywhere Cloud and it’s been in tech preview with big customers for a few months and it releases next month. And it is, with all due respect, a bit of a game changer because it hits the needs that we were just describing.

Swapnil Bhartiya: First of all, thank you for explaining that. The journey as well. Now, as the main theme of today’s discussion is of course the distance between where AI is and where data is, let’s also talk about why most AI pilots fail. When an AI pilot tries to become a production system, where do things start to break? First? The data, the governance, the cost of running it. Problem is not the models, problem is somewhere else.

Leo Brunnick: Yes, correct, correct. I like to say that the first issue is almost always governance because for AI to work, it needs to access all of the data that it needs. But there’s problems with that. When you think about a person trying to access data, it’s pretty simple. Because is this person supposed to be able to see the CEO’s salary? Probably not. Okay, so you have rules, you have roles, you have governance inside your company for role based control, for, you know, by person control and fine grain control down to different pieces and parts of tables and databases and etc. And that’s just been, that’s just a fact of running companies for the last 315 years. But now when you get to AI and you get to agents, it becomes just a very different game, right? Because for an agent to be successful, if it’s a administratively created agent like in has to be able to access everything the company has and then the user of that agent can only see things that that user is allowed to have. But if the user creates an agent, an agent can’t be created at a higher access than that person has. And people start struggling with all this stuff and they, and they’re just not ready for it. They don’t have the guardrails in place. And that is something that makes something that looks really cool in a sandbox really hard to deploy across most corporations. So governance is the first one, right? And then simply data access, right? Because if you work in a company that’s most people have a messy vendor estate and a messy data estate, right? So you can go out and do a project and then you can manually go out to SAP database somewhere and get some other stuff and you can kind of mash that together. Telling AI to go do that isn’t as easy as it sounds, right? In other words, all of this thing, both the governance and the access, comes down to, you know, the my. One of my favorite words is data fabric, right? You’ve got to have a unified data fabric and that is easier, easier to say than to do. And then of course the final one, you already said it’s cost, right? Because to do something, you know, a thousand times might seem like, hey, that’s no problem. But when a thousand people do something a thousand times, suddenly it is, you know, a hyperbolic curve of cost that tends to shut projects down unless the ROI is so smack in your face obvious it tends to be pulled back. So it’s really all three of those things wound together that make putting AI into production complicated for most companies.

Swapnil Bhartiya: As you rightly said, enterprise data doesn’t sit in one place, it could be anywhere. Data center, multiple public clouds at the edge. Now your approach, as you rightly said, is to bring the AI to the data rather than move the data where the AI is. When a customer tries to centralize everything. What starts to break because once again, things are not as simple. AI has made things very, very complicated. So what starts to break because what I’m trying do is to break down the problem areas, then we’ll focus on the solution area.

Leo Brunnick: Yeah, so getting all of the data to data is heavy. You know, moving data say, okay, I always use the phrase data gravity. Data is heavy. Trying to get data all into one place is very difficult. If that place is, you know, on premises in your data center, you’re not taking advantage of what the cloud can bring you. Clouds are just, they’re essential for being able to do certain things in, you know, the corporate, you know, corporate care abouts. By the same token, if you try to move it all out to a single place with a single vendor on a single cloud provider, that creates a different set of issues. Right. Because you have, you know, all sorts of costs to get it moved, massive egress costs. If you want to do some other things, there are things that you simply must do. Every corporation in a private environment, which means I now have two copies of the data and it’s back to not being in one place. So you’ve got to accept the fact that the data is going to be in a number of different places and you have to have the infrastructure that allows you to connect all that data as though it were in one place.

Swapnil Bhartiya: Now if you look at your great rearrange survey, can you talk a bit about. Of course, you know, a lot of respond, almost 84% said that AI workloads have pushed their infrared cost up and that is becoming a big problem. That’s why we are seeing the whole tokenomics discussion these days. But if you can just highlight some other findings which are becoming recurring problem and which are also not the problem. Which are the some problems are whenever new technology we have seen with Kubernetes with Docker container, as the technology matures, some problems will automatically go away. But here these are the problems which I sometimes feel is not actually even sustainable. So I want to talk about this survey some of the core findings which will also help us further expand the problem. And then we’ll start talking about partnership and how you’re trying to solve this.

Leo Brunnick: Yes, I think unsurprisingly we find that companies end up spending more than they anticipated. Right. We hear all the stories. It’s February and I’ve used my annual estimated budget in cloud token spend. You know, it’s Monday and there goes the week. Right. Because we just, you know, ran. I don’t think people realize the mathematics when they start using agents for what kind of accumulated tokens the systems do, right? And as you said, experimentation phase, you know, everyone’s going to get better at it and etc. The problem is for the, for the cloud vendors, the demand is still outstripping the supply, the demand for hardware, the demand for processing power, etc. So there is no cost abatement yet, as a matter of fact, it actually just continues to get worse and which is causing companies to then look much harder at each project for approvable ROI before they’ll allow it. Okay, you know, and that’s another one of the findings. You know, the dollars are so real that getting to a what I’ll call provable ROI becomes exceptionally important. All the big companies we work with are investing in on premises GPUs, GPUs that run under their control, where they have a budget, they spend it. And now they can work on a predictable model and run with amortized capital costs instead of out of control operating costs. And every large company that I could name has that, at least as part of their model. And what I find fascinating is that in a survey last year, JPMorgan Chase survey last year, the aggregate CIO, the CIOs in aggregate said that they’d be doing between 35 and 45% of their inferencing on prem going forward. And at the time of the survey there were zero on prem inferencing solutions. But the market was so in demand that everyone knew it was coming. Right. So I think what you’re seeing now is a pivot to that becoming more and more real.

Swapnil Bhartiya: Let’s get a bit more info about your partnership with Nvidia. The whole Cloudera inference, because it now runs on blackwall, Dynamo, Teton, I mean all the hardware, and also using opendisk like Nemotron. Talk a bit about this partnership and the importance of this partnership for Cloudera and for the ecosystem and for the customers.

Leo Brunnick: We partner with Nvidia in a number of levels. Cloudera was the first vendor last year to make generally available an on prem inferencing solution. And we’re using Nvidia’s, Nematron and et cetera. There are technologies that allow us to do inferencing. We are doing the next generation that won’t talk all product, but the next generation of that releases next month. And we’re a pretty tight partnership with them on a whole series of levels. For example, we work with them on because Apache Spark, as you know, runs a lot of enterprise data pipelines and our Cloudera Data Engineering, which is our data service that has a number of pieces in it, but it’s centered around Spark in order to do data engineering. And again, because it’s a containerized data service, it runs in any place you need it to run, but it has native GPU acceleration built into it. Right. So working with Nvidia’s libraries, they have names like Cuda X, but you get up to four times workload acceleration, which means that you are more efficiently using your resources and you’re therefore spending less money in order to do the things that you need to do. You know, again, if you can be doing heavy data processing on premise, you’re already saving money in that you’re not paying for accelerated cloud costs when you do this. When work with Nvidia to get up to four times better performance, that again brings the cost down and down and down for all the things that you need to do. Which, you know, suffice it to say, customers are looking hard for anywhere they can, they can spend less on what they’re already doing because they’re spending so much more than they anticipated.

Swapnil Bhartiya: In some other areas, infrastructure is there, models are there, agents are there, governance is there. But before model can run, data has to be prepared as well for that. And if you look at Spark pipelines and other approaches, they can take a lot of time, they can take hours. Of course the cost pile up again. Can you talk a bit about how does Cloudera help with getting data prepared with of course, the whole GPU acceleration with Apache Spark through Nvidia’s and everything else. Because time in this era, latency leads to cost.

Leo Brunnick: Yes, yes. And we mentioned earlier that you have to bring AI to the data, right? And that is about, like you said, both cost and about security and it’s about a lot of things. But the word latency really triggered something for me there, right? Because there are people out there that are cloud based and they’ve always said bring all the data to me in the cloud. And in the last couple years, people like I’m now, I’m not, I’m not doing that, I’m not doing that anymore. I’m not making copies of it, I’m not moving it back and forth, I can’t put stuff out there, I’m not bringing it. And so those same cloud vendors are saying, okay, I’ll stay out here on the cloud, but I can do your data work or your AI work for you from here. In other words, I’m now hybrid. That’s not hybrid. It’s not hybrid. It’s not hybrid because of the latency. You actually have to bring the processing, bring the inferencing, bring the AI or the, in this case, you know, data processing to the data. Right? Because latency is real. I mean latency is the currency of the, of the, of the AI era, right. So you asked how Cloudera helps with that. We’ve always been the, we were the on prem company forever, right? So you know, when we said, originally said hybrid, it was because we were on prem, we could do a little bit of cloud. And then as we’ve moved forward over the years, it’s become so true. You know our, we have something we call anywhere cloud. It’s our, it’s our way of doing data and AI platform. But the anywhere cloud is the same identical piece of code, right? The same self contained AI platform that sits on Prem or on any one of the hyperscalers as an identical piece of code which allows people and then we tie that together with a unified data fabric. So it all works as one platform. And that’s when you start getting into the ability to do the speed and the innovation at controlled cost with the governance that you need. And those are, that’s a, that’s a tough set of things to pull off all at once. So that’s certainly how we’re trying to address the problem for folks.

Swapnil Bhartiya: What does a model refresh look like in a distributed deployment where the same workload is running on prem in the cloud at the edge? Because the way you have done the whole agents, prompts, models has changed so outcome also starts to change. So it’s kind of, you’re dealing with the added complexity. So talk a bit about that aspect.

Leo Brunnick: We try to be agnostic around a series of things. For example, people have to store things in an object store, right? So we do all open standards, S3A. So you can use Dell or Vast or Minio or NetApp or whatever you want to use. When it comes to models, we want people to be able to use whatever models work for them. So our LLM gateway is exactly that. Right. I don’t care what model you’re pulling off the hugging face or whatever else it is, you can use any model. With our system, with our inferencing and we recently, and this was about a month ago and we announced a partnership with Mistral. Mistral is interesting. Mistral is, as you know, it’s a European model provider. And the difference between them compared to let’s say anthropic or OpenAI is that their whole game is open weight models, right? So they will provide you a model and that allows us to run it on prem and run it inside our system, wherever it might be, including on prem, in such ways that it solves for people the problem of private AI and sovereign AI, especially for all these big European customers. Because handing someone an open weight model is the same as handing them a piece of on premise software. You’re handing them the code. Right? Which has become a really big deal. Big deal of financial services in the US Big deal for any international corporation with businesses in Europe because of sovereignty. Which is why I’m so fired up about the Mistral side there. As you know, these frontier models, everyone’s pouring just so much into them. It’s a 10 billion, it’s a trillion, it’s a this many, you know, you know, weights in the system, but over time that asymptotically get closer and closer. Right. So are you willing to pay 100 times as much for something that’s 1% better? That’s becoming the mathematics. So a company like Mistral is, has gotten a lot of attention because they are, you know, maybe the third best, maybe the fourth best frontier model, but they’re open weight and they’re going to hand it to you. And that is really, really fascinating to people. We also are working with our first customers on doing model training, Cloudera, Mistral and the customer to do because they’re doing, you know, specific models for those companies we have, I won’t name them, but a couple of financial service companies whose names you’d know believe that that for them is a big deal. To have the model that they train to get the value that they want out of their hundreds of petabytes of unique data on prem is just to them there’s a lot of zeros of value in that for them.

Swapnil Bhartiya: No. Very well said.

Leo Brunnick: And so as, as people have started challenging tokenomics and token maxing is like, yeah, I can’t do this anymore. We have one customer. I won’t even say what industry. I don’t want you to tell you where they are. They were building out a big data center, but in the meantime aggressively leaning in the words they said to me 18 months ago were, look, I know it costs a lot of money and I know a year from now or two years from now I could get fired if I’m over budget, but I’ll get fired next month if I can’t prove I’m moving these AI projects forward. In other words, cost is way down here, fast forward 18 months, where their annual cost went from 30 million a year in their token spend with a particular vendor on a particular cloud to 900 million a year in the span of, you know, just over two fiscal years, it went up 30x. That’s real money. And no matter what business you’re trying to run, that doesn’t go on forever. So the bloom has come off the rose on some of that kind of stuff. And so if you’re OpenAI or you’re anthropic to justify your trillions of dollar valuation, you said, look, it’s not just about these tokens. It’s about the fact that our models continue to learn as people use them and they get better and better and better, and we can start doing things that used to be done by ERP vendors, that used to be done by this out of the other thing. And so they essentially said, people use our models, we learn their business, and then we can do their business. And that’s when a lot of the, you know, CEOs were like, oh, timeout, timeout. I’m not putting my Alpha out there anymore. And like, so Europe’s all getting concerned about sovereignty because of world events. US is getting concerned about private AI, that version of sovereignty, because they don’t want to give away their, their Alpha. And between those two, they’re ending up at the same place, which is they must be able to do quick innovation, easy deployment, rapid changing of models, etc. All that kind of stuff. And, and you have to be able to do it inside your firewall. And that’s going to be the play going forward. We haven’t even talked about edge and far edge of down to handhelds and et cetera. But the AI, wherever the data is, and the AI close to where the processing needs to be done is becoming the way of the future.

Swapnil Bhartiya: No? Very well said. Thank you for getting in detail there as well. Which also is a perfect segue to the teams who are watching it, who have a pilot that works, but they are not sure whether it will reach in production. What’s the first thing you would advise them to fix?

Leo Brunnick: The first thing that I would advise them to fix is it depends on what kind of project and what kind of company. Because you can go after cost, you can go after governments, and you can go after access to data, but for everybody and what we see in our customers, the one that derails projects over and over and over is around governance, right? In other words, this, this agent can do a lot. My kid could create an agent that can do a lot. That doesn’t mean it should do a lot. Right? So it is, it is the, the, the operative word heading into the next couple of years, and you hear it all the time is harness. Okay? The value. Data has value, models have value. But in between, it’s the harness, right? It’s the thing that the, that the AI gets deployed within. Right? And it is, you know, I use their term governance. Governance. Governance. But you know, that’s where that word might get misinterpreted. Your corporation’s series of rules for who is allowed to see and work with what data is the key. And any project that’s being built, any agent that’s being built has to fit in within that framework. In other words, an agent and a person have the same requirement to be governed by that set of rules. And here’s the, here’s the cool part. Most big companies have those rules already. Most big companies, the ones that have these big data centers, they have those rules in place. And the problem is when people try to go do these AI projects, they’re taking data out of the data center, they’re, they’re putting it out in the cloud, they’re running agents against it. It’s like, whoa, wait, whoa, whoa. Because my rules didn’t go out there. You know, my rules, my governance didn’t go out there. Which is why if you can have a unified data fabric that allows you to do all that stuff on prem, or travels with the data, or travels with the AI and the agent to a cloud, now suddenly you can actually do production things with your AI. And it’s not that hard a leap for an organization to get from the human based rules they have to a data fabric that will work with autonomous agents. It just, it’s a change in thought process and having the right piece of software to implement it.

Swapnil Bhartiya: Leo, thank you so much for this. It’s clear that by now, it’s clear that what actually, what it actually takes to move AI from a pilot to production. Your advice is really valuable. So viewers who are watching this, please, if you want to dig deeper, go and check more about the work that Cloudera is doing along with [email protected] and Leo, once again, thank you so much for your time and I would love to have you back on the show.

Leo Brunnick: I would love to come back. It’s fun talking to you. Thank you.