Swapnil Bhartiya: It seems that the enterprise AI honeymoon phases over and organizers are quickly realizing that renting intelligence, by the token, can be an expensive, unpredictable way to run a business. And to unpack how companies are taking back control of their AI infrastructure. Today we have with us Christian Stano, field CTO at Anyscale. Cristian, it’s good to have you on the show.

Christian Stano: Thanks so much for having me. Really, really excited for the discussion.

Swapnil Bhartiya: It is a very important discussion. Before we dig into the shift towards sovereign AI and what it means for enterprise customers and even for Anyscale, I would love to know a bit about the company itself because this is the first time I’m talking to you folks. So tell me a bit about the company, when it was created, why it was created.

Christian Stano: Anyscale was started about 10 years ago. Ray is a open source project that kind of kickstarted all of this. It’s out of the UC Berkeley lab. So our founders, Robert, Ion Stoica and Philip essentially faced a problem as part of their PhD research, where they quickly discovered it was actually harder to do the research because they were spending all their time solving the big problems around distributed AI systems. And so they created this open source framework called Ray. In the last 10 years there’s been steady growth and in the last couple of years with the invent of transformer models and LLMs, the growth curve of Ray is skyrocketing. So now we’re powering over a million unique clusters per week across thousands of different companies and enterprises from companies like Uber to Apple to Cursor to X. If there’s a foundation model being built, there’s a good chance that it’s being built by Ray. So Anyscale is essentially the company that is taking the open source, supporting it and then building a lot of the governance and enterprise grade support around that experience so that when the larger customers come to us, we can provide them that open source framework with security, the stability and the scalability of an enterprise grade software platform.

Swapnil Bhartiya: A lot of enterprises, they kicked off their AI journey just by plugging into some hosted foundational model APIs. What you’re hearing on the ground today as AI has moved out of experimentation pilots to actual production workloads and also today AI is not just simple chatbot which is answering your questions, agents are taking actions on their behalf and sometimes it can get stuck in a loop cause cost may pile up, you will have no idea. Talk a bit about what you’re hearing on the ground where customers are kind of rethinking and they want to kind of refine or own their AI infrastructure.

Christian Stano: Yeah, it’s a great question. And it’s funny, on my commute into the office this morning in the Bay Area, you look at the train and every AI company right now is broadcasting own your intelligence. Own your intelligence. Like what does that actually mean? And so what I’m hearing in the field with some of the largest customers building these business applications that are powering business critical workflows is it actually is a very logical first step to start with the API providers. You want to move quickly, you want to get off the ground, you want to prove that you have business ROI as part of this investment. And that’s what we’re seeing a lot of customers is we’re seeing, hey, there’s ROI here, there’s business value here. But on the flip side, my token costs are skyrocketing and I’m also thinking about the supply chain, I’m thinking about the political landscape, I’m thinking about the business reliance on a lot of these API providers. And that’s where I think we’re seeing a lot of these customers shift towards owning your intelligence instead of renting. And I think the cloud reference is a really good one. There’s a paper from the late 90s called what Goes around comes around. And I think we just see these cycles in technology that kind of keep repeating from rent to own, rent to own. And so I think what we’re seeing and what I’m seeing with a lot of customers that I’m working with is starting from that inference layer. It’s becoming more and more important to own that component so that you have a scalable and stable cost line item as part of your CapEx and OpEx budgets for a lot of COOs, CFOs and CTOs. Um, so that’s kind of the, just like the tokens per second and tokens per dollar that I’m getting on these business critical applications. But that’s just the first part of the equation here because there’s a whole host of things that kind of come to the left and to the right of that inference component that I’m seeing a lot of customers start to think about two specific areas that come to mind. One is the post training world. So as you kind of mentioned before, these open weight models are getting better and better. What I’m seeing in the field is that customers are fine tuning those models with their data to be able to build fit for purpose fine tuned models that then plug into inference. So then you’re starting to control the token costs. Now you’re starting to control and fine tune the model for performance. And then the last piece of that equation is you’re starting to think about ownership and sovereignty is the data. 90% of these systems are actually data and just different ways of using that data. And so what we’re also seeing is a lot of enterprises continue to move even further left into owning those data pipelines. And then the question that that opens up is how do you think about that end to end pipeline and what does it take to actually solve that in a stable, scalable, price performant way? That’s where our recent release of Anyscale on Azure comes into play. For enterprise customers running on Azure, we’re able to kind of help them own that end to end sovereign pipeline. But this is a pattern that I’m seeing across both the Best and Frontier foundation model labs to some of the largest enterprises in the world is I need to own that inference for cost, but from a longer term strategic lens, I also need to own what is that model doing from a fine tuning perspective and then how do I actually activate my data to then push that through the rest of this pipeline.

Swapnil Bhartiya: Looking at the geopolitical situation, looking at all the conflicts that are going on, looking at how businesses evolve, what are the real when it comes to sovereign AI or owning your own AI, what are some of the real components that you think these are the ones that people should focus on because rest are wearables, they can keep moving around. Does that question even make sense?

Christian Stano: Yeah, it makes total sense. And what’s really interesting right now is we run an international business and depending on the kind of geographical zone of the business, the answer to this actually changes. I think what’s absolutely critical to think about from an ownership perspective is threefold. One is do you own the inference and the system that your business is running on? And there’s a huge component to that of stability and the cost line item that we just kind of discussed. The second piece, which is I think we’ll see more and more as we go into 2027, is do you own the actual model and the model weights of the thing that’s actually providing that inference? I think you can make the argument of swap this out with a foundation or Frontier Lab, build your own, have the ability to be flexible there, but it’s really critical to own that component. But the most most critical component to own is the data. When companies are investing billions of dollars of capex into GPU infrastructure, not having control over where your data lives and where it goes as well as the model weights that that data informs for your actual inference endpoints is absolutely critical as we move into the next year. And even beyond that, when you start thinking about like I just made this huge investment, what is the security and compliance service area that I’m opening up by not owning that for a lot of enterprises, that’s like a day zero, no go on renting, you have to own that. Depending on if you’re in regulated industry, like in banking or healthcare, that’s absolutely critical. And then the other component is as CFOs and CTOs have to answer the question of, hey, I just spent tens to hundreds of millions or even billions of dollars on my infrastructure and my hardware layer. How do I guarantee that I’m both utilizing that, but also justifying that investment? And the way to do that is saying, hey, I own everything end to end from a security lens, from a performance lens, and from a utilization lens. And so when we start thinking about sovereignty, that is the actual really critical piece is when you start tying together those three layers of the inference endpoint. What my business runs on, the model weights, what my business is essentially dependent on, and the data, which is how I actually activate the business value that I’ve been accruing over the last dozens or tens of years. That’s where really the sovereignty piece comes in, paired with the secure security and compliance lens of hey, if I rent this out, sure I get assurances, but I’m essentially limited and I have a ceiling on what I can then go and tackle as a business or how much I can actually provide without risk of exposure or increased security. Surface area.

Swapnil Bhartiya: Excellent. Thank you. Now the, the challenge with AI, of course the best is you run your models locally. But the thing is hardware availability, their own cost, and then whole plumbing that it takes may not be worth it in most cases and it will be counterintuitive either way. New models coming, you are getting it. The cloud does play a big role and you folks also announce any scale on Azure to kind of let organizations build and run AI systems entirely within their own cloud environment so they do get that control that they want. Can you talk about what are the practical day to day advantages of doing this, especially for those highly regulated industries? And you mentioned your clients are globally as well.

Christian Stano: This is something that is very near and dear to me because I spent close to a decade in govtech defense tech and in regulated industries before I came to anyscale. And there’s two components here to think about. One is what is the box that I’m drawing around those three components of sovereignty that are required, like the inference, the models and the data and do I own all those things? And then the second piece that is a little bit lesser talked about is as I go from kind of these experimental proofs of concept to actual production grade AI systems, what does that look like and what does it take to build and maintain that system? And so the Anyscale on Azure release really helps both of those fronts. One being if you’re an Azure customer, we have the first party Entra ID integrated security and identity that kind of comes along with running inside of Azure, just already baked into Anyscale. So that’s kind of helping with the sovereignty and the security piece to it. But the other piece that we see with a lot of customers and that when I was running my AI infrastructure team we faced is there is this tax that exists when you start owning production grade AI systems. You essentially in the world, without any scale are stitching together all these different components of the system, from data preparation to model training, to model fine tuning, to deployments, to serving and the online inference components. And without Anyscale, you’re essentially using point solutions for each of those. Which works if you have a huge team and have already invested into that. But with where we’re at in the industry, most folks have not done that yet or that burden is too high from like a operational standpoint or the timeline to do that is much too long. We’re typically seeing customers spend anywhere from nine to 18 months building those types of systems to just get to their first workload and so on both those fronts, that’s where Anyscale is really making huge differences. I want to leverage the advantages of the security and compliance posture of Azure because I’m running in a regulated industry and I need to move as quickly as possible because my business relies on this and I’m just watching my token costs go through the roof. And so when you put those together, you have this kind of perfect confluence of I can go from experimental POC with API providers to production grade pipelines in under six months. And that’s extremely, extremely valuable for folks in regulated industries or in compliance environments who are looking to kind of balance that act of I need to move fast, but I need to be secure.

Swapnil Bhartiya: When it comes to enterprises. And as they plan to move away from those unpredictable per token cost towards infrastructure that they actually control, how should they be the trade offs between renting intelligence versus actually building AI capabilities as a core static asset? At the same time, how should they also rethink their AI architecture, their AI infrastructure itself.

Christian Stano: Very topical. I actually just spoke with a couple of CTOs of some of the larger banks in the US in the last couple of weeks on this exact topic. And so I’ll start with how you should be thinking about this problem and then we’ll kind of talk about how that influences the architecture component to it. So there’s three things that every CTO should be thinking about as part of this. One is the price, performance and speed. Trade off. Going from an API provider to owning your intelligence, you are going to be constantly facing a trade off of speed versus price. You’ll be improving your price over time long term, but you’re going to be moving slower because you essentially have to invest the months of work to stand up the stack before you can start switching those workloads over. So there is a kind of ROI curve that you need to evaluate against the important milestones to your business. And so there’s always a constant trade off there. The second piece is do I even have the people to do this? That’s something that is a much, much harder problem to solve than a lot of people think because there’s so few people in the industry who can do this successfully. And on top of that, those people are very expensive. And so as you’re thinking about the headcount budget and the actual team dynamics to build these systems, that’s a huge component to be thinking about. The third one is going into kind of the supply chain layer of in order to build these things, I essentially have to be thinking on the scale of multi year commits with a lot of the hyperscalers or the Neo clouds or on prem providers for GPUs because you cannot only systems unless you own the GPUs as well. And so there’s a huge component there of have I started those conversations and am I timing those correctly? Because what I’m seeing in the industry quite a lot is if you’re starting now, you’re likely looking at at least six to eight months before you can even procure some of those GPUs. And so then you are essentially time boxing how fast you can move. So there’s those kind of critical pieces of the speed and cost trade off the people to actually do it. And just can I get the hardware that a lot of these kind of architects and companies should be really thinking about right now in order to land this in 2027? From the architectural piece to this, what is great is that there’s a really natural evolution from today’s kind of traditional machine learning system into these LLM based stacks of the future, where a lot of the operational processes and pipelines look the same, the tooling is just different, the scale is different and the cost is different. So what I’m seeing is the traditional kind of ML pipeline of data prep to training, to tuning to serving still exists. A lot of the operational processes still exist. What’s changing is that now you have these incredibly expensive GPU fleets, so you now have to think about that variable as well. And then these models are getting larger and larger and larger, both from the inference side and from the training side. And so that adds a new layer of complexity as well of how do I distribute that across my compute estate really efficiently. And so from an architectural perspective, as you’re making those trade offs on those first three that I mentioned of cost and speed, headcount and the infrastructure, you are essentially taking your existing processes and sort of evolving them and up leveling them to be able to meet the these needs of tomorrow with your own LLM sovereign AI stack.

Swapnil Bhartiya: As we’re talking about Ray in the beginning, which is already hugely popular for scaling AI workloads within the open source committee, where does Ray fit into any skill on Azure? And why is having this unified compute layer so critical? As AI is becoming of course multimodal and agentic, and of course you have to look at what owning IT and whole server in part as well.

Christian Stano: I’ll start with the use case component and then I’ll kind of move into the architectural component. So what Ray does really, really efficiently is as I’m building these different types of workloads and use cases to power my sovereign AI stack, Quite commonly you are mixing CPUs and GPUs and that’s actually the really efficient way to build a lot of these pipelines. Because GPUs are a scarce resource, I want to use them for my most important workloads. So anything and everything that I can offload to a cpu, I want to do that. Ray helps you do that in a really, really efficient manner, from data processing to training and inferencing, so that I can manage these fleets of compute in one large pool. On top of that, Ray has a huge amount of fault tolerance, failure recovery, observability that’s baked in alongside its Python developer friendly APIs that make it a really developer friendly tool to use. So as you’re thinking about building the stack and you’re choosing that framework to build it around, Ray checks a lot of the boxes that are required to go from something like a RAW video file to an autonomous vehicle and everything in between. So you can kind of build your entire pipeline on top of Ray without having to pay down those different point solutions. Then architecturally, where that fits into Anyscale and Anyscale on Azure is it fills this runtime layer that sits between the developer experience, kind of where our developers are interfacing with this platform to build those different components and the hardware layer of my GPUs and CPUs on Azure or other cloud providers. So Ray is kind of what powers the engine and then Anyscale provides an enterprise grade developer experience around that. So that when I’m thinking about hey, I need to give my developers access to this massive fleet of GPUs, I need to test out my inference endpoints, I need to fine tune a model at a really large scale. I can do that in a really predictable, really efficient manner. But Ray is the backbone in the central nervous system for all of that.

Swapnil Bhartiya: Can you also talk about how is the climate around AI changing? Especially what happened with Fable and before Mythos. Organizations are also skeptical of putting all of their eggs in the same basket not knowing that the model that they’re they, I mean a lot of folks, they just build the whole things based on and if that rug is pulled out, think about it. Suddenly it’s like you five somebody pushed your five developers, you have nobody left to work on that. What kind of climate sentiment you are hearing in terms of open source? Just the way open source, the Linux kernel, Kubernetes, OpenStack, it has democratized a lot of other spaces. And the fact is that some of these open weight models they are very, very close to the front end models. But the thing is I mean that we have seen it open source sometime when it comes but what are we hearing where the whole climate change organization are starting to think more about open source as well in AI space?

Christian Stano: I think what’s really interesting about this conversation is that if you rewind like six months we probably wouldn’t be having this conversation. I think that’s a testament to how fast the space is moving. I think that’s what makes this such a challenging question is these Frontier foundation model labs are building fantastic models and there’s a huge advantage to going with like an out of the box highly intelligent API. But at the same time the open weight models are catching up or with techniques like fine tuning and post training you can then cater those to the specifics of your business. And that’s a Huge advantage that a lot of companies are starting to take advantage of. In this kind of overlapping climate of renting has this potential instability or vendor lock in. At the same time we have these new techniques where we can actually take these models and apply them specifically to my business using my own data to eke out gains in performance that I would be able to get with an off the shelf model. And so within this climate, I think we’re seeing this combination of the two. From the technical lens of hey, now I can actually do this with an open weight model. And from the political lens of I don’t know what’s going to happen in the next six to 12 months. Obviously these foundation model labs will continue advancing their models towards superior intelligence and there’s a huge advantage to that. But kind of in the classic build versus buy scenario, a lot of engineering leaders and product leaders need to be thinking about what are the implications and the risks of that rent versus owner conversation. And I think it’s very tailored to each business. I know a lot of businesses who are extremely happy with renting and kind of using those off the shelf models, they built a huge amount of harness infrastructure around it and a lot of context for that. And I know a lot of businesses, particularly in the European regions and Asian regions who are leaning towards I need to own this end to end because this is my 5 to 10 year strategy for AI. And so I think we’re seeing this combination of the two.

Swapnil Bhartiya: Ruchy and thank you so much for joining me and sharing these great insights. It’s true that getting the economics and infrastructure right is clearly going to make or break the next wave of enterprise AI. So thanks for the work that you folks are doing at any scale and it was great chatting with you and I look forward to chatting with you again. Thank you.

Christian Stano: Likewise. Thanks so much for having me.