AI Infrastructure

How to Choose Workload Placement in the AI Era | Bryan Litchford, Rackspace Technology | TFiR

0

Every enterprise has to decide where each workload runs, balancing performance, governance, cost and latency. Once AI agents join production workflows, that choice also decides how AI infrastructure connects to core applications and data, and how much data has to move between environments. There is no single location that fits every workload.

In this interview on TFiR, Bryan Litchford, Head of Product, Engineering and Data Centers at Rackspace Technology, walks through how enterprises weigh data location, governance, latency and cost, where to place AI agents, and what to test before moving a production workload.

Guest: Bryan Litchford, Head of Product, Engineering and Data Centers at Rackspace Technology
Show: The Agentic Enterprise
LinkedIn: Bryan Litchford
Watch the discussion on YouTube

Here is what every platform engineer and enterprise architect needs to know.

Technical Deep Dive

Q: How should enterprises choose where each workload runs?

Bryan Litchford, Head of Product, Engineering and Data Centers at Rackspace Technology, says this problem is 10 to 15 years old and the answer is to unpack each workload instead of applying one solution to all of them. Latency-sensitive and data-sensitive platforms, such as healthcare systems where patient information and uptime come first, often belong closer to where the work happens, for example a data center in the hospital’s city rather than a hyperscaler region. Since COVID, when workloads headed to one hyperscaler, Litchford sees a healthier move to hybrid cloud for them.

“It’s really important to unpack each individual workload as opposed to just creating a one-size-fits-all solution.” Bryan Litchford, Head of Product, Engineering and Data Centers, Rackspace Technology

Q: How should AI agents connect to core applications and data across environments?

AI agents add urgency to knowing where data and applications sit. Litchford expects GPU-enabled agentic workloads to sit next to core business applications, for example inside bank branches where instant response is expected. He also sees more adoption of smaller language models for well-defined workflows: one document-extraction customer moved from a frontier model to a tuned small language model placed next to its document store, and processed everything locally at lower cost.

“So instead of sending traffic from one cloud to the next to process, we did it all locally.” Bryan Litchford, Head of Product, Engineering and Data Centers, Rackspace Technology

Q: When does placing AI infrastructure close to production applications reduce latency and data movement?

A data store is not directly useful to an AI agent, so a RAG application needs a vector database between the data and the LLM. If the agent runs on a frontier model or hyperscaler, Litchford says, data must be fed to that cloud, or the application becomes chatty over an API, which raises security and latency concerns. Putting the agentic system next to the core system, even on the same switching infrastructure, removes both and keeps the footprint hosted end-to-end in one place.

“That eliminates your latency concerns and gives you greater control because you’re hosting that entire footprint end-to-end in one place for that particular workload.” Bryan Litchford, Head of Product, Engineering and Data Centers, Rackspace Technology

Q: What changes when VMs and Kubernetes run on the same platform?

Customers no longer want to lift and shift everything. They want core VM or bare metal workloads to run next to newer containerized applications. With VMware Cloud Foundation 9, VMs and a managed Kubernetes environment run on the same hypervisor infrastructure and a single control plane. That reduces what Litchford calls hardware bloat from separate Kubernetes and virtualization environments, and it allows workloads to move between locations.

“What we found is customers don’t necessarily want to just lift and shift anymore.” Bryan Litchford, Head of Product, Engineering and Data Centers, Rackspace Technology

Q: Where does VMware Cloud Foundation end and the provider’s operational responsibility begin?

Rackspace is responsible from the data center up to the packet leaving its network, and VMware Cloud Foundation 9 is one core component of that service. Beyond bringing the platform, Rackspace manages hardware procurement and supply chain, warranty and replacement, failures and the data center itself, at a time when data center space and power are hard to get. Almost 30 years of operating data centers also informs the design of each customer’s footprint.

“Rackspace, as I like to say, is responsible for everything from the dirt under the data center to the packet leaving our network and traveling to an end user or the public internet.” Bryan Litchford, Head of Product, Engineering and Data Centers, Rackspace Technology

Q: What should you test before moving a production workload to a new cloud platform?

Start by understanding what you are moving and why: tech debt, resilience, performance or a more modern landing zone. Then map what the workload interacts with and whether it is a monthly reporting job or operationally critical, because critical applications need disaster recovery and cyber resiliency designed in. For core applications, Litchford’s checklist includes the 99th percentile, the absolute peaks and valleys, plus location concentration and disaster recovery.

“I think what’s more important is understanding which workload you’re moving and why, and building your test plan around that.” Bryan Litchford, Head of Product, Engineering and Data Centers, Rackspace Technology

Resources & Documentation

  • Multi-Tenant Private Cloud, Rackspace’s multi-tenant private cloud offering
  • Enterprise AI Cloud, Rackspace’s cloud offering for enterprise AI workloads
  • GPU-as-a-Service, Rackspace’s GPU infrastructure service for AI workloads
  • VMware Cloud Foundation 9, the private cloud platform Rackspace has selected as its core platform

***

👇 Click to Read Full Raw Transcript

Swapnil Bhartiya: As we all know, it has been a holy grail in the IT world that choosing where a workload runs means balancing performance, governance, and cost while keeping applications as close as possible to the data that they need. And in the age of AI, it is becoming even more critical because as enterprises bring AI into production, those decisions also affect how agents connect with core business application, how to deal with latency, how to deal with governance, harness privacy. A lot of factors start playing there. The challenge is choosing where each workload belongs without creating more data movement or more systems to manage. There is no one-size-fits-all. There is no one basket where you put all your eggs. And joining me today to discuss all these is Bryan Litchford, Head of Product, of Engineering and Data Center at Rackspace. Bryan, it’s great to have you on the show.

Bryan Litchford: No, thank you. Thank you for having me and Rackspace and letting us join into the conversation.

Swapnil Bhartiya: As I was talking about the whole, I mean, this is not a new problem, but AI has kind of latency. Of course, cost is a big factor as well. When enterprises decide where a workload should run, how should they weigh data location, governance, performance, and cost, and we can also throw latency there.

Bryan Litchford: It’s a problem that organizations have been grappling with for, you know, 10, 15 years really, and even before that, before most businesses were, you know, completely digitized. When, if we look back, it feels like so, so long ago, but if you look back at just during COVID it seemed like all workloads were headed in one direction, right? They were headed to a hyperscaler and they There was no consideration to the types of workloads, the types of services that those platforms might need, or the maybe particular governance or control that’s required. And so what we find is, as we’re working with enterprises, it’s really important to unpack each individual workload as opposed to just creating a one-size-fits-all solution. We spend a lot of our time talking to organizations about latency-sensitive platforms, platforms where data is of the utmost concern. Rackspace supports a ton of organizations in the healthcare sector, and you can imagine patient information is honestly this, like, the top priority right there alongside uptime of these systems. And it’s important that not only does data live in the right place, it has the right controls around it. Right. And system access from other applications is governed quite a bit. And so what we’ve seen since COVID where everybody was sending everything in one location, we’ve started to see this kind of healthier, I think, adoption of hybrid cloud solutions for big enterprises with some of those latency-sensitive workloads. We see them bringing that closer to where the work is happening. In the case of a hospital system, you know, a hyperscaler region might not be the closest location for them. It might be a data center in the city. that their hospital system operates. And that latency matters. It matters for physicians, matters for nurses, matters for technicians doing X-rays and things like that. And so there’s these real-world effects based on the particular workload where it should live.

Swapnil Bhartiya: Of course, we touched upon that, and that’s becoming a huge discussion these days. When AI agents become part of a production workflow, how should enterprises connect them with core applications and data that run across different environments?

Bryan Litchford: I think this is— AI agents have just added a layer of urgency in understanding where data and certain applications sit. I see this future where agentic workloads, AI-enabled workloads, could be systems that are only CPU or probably more realistically GPU-enabled. Yeah. that sit right next to some of the core business applications. So for a bank system, right, they’re going to expect that some level of their interactive system is not going to be dependent upon a core location that might be in one particular region inside of a country, but instead they’ll look for some of these systems to sit inside of bank branches, for example, right? You want that instantaneous response. And I think the other thing we’re seeing is Models have changed quite a bit over the last, you know, it changes weekly, it feels like, but we’re seeing more and more adoption of smaller language models for really well-defined or automatable workflows. Uh, organization we’re working with that does a lot of data extraction from documentation, they were running thousands and thousands of documents through one of the big frontier models. And really what they needed was instead of paying for all of those tokens. They needed a much more optimized setup. We took a small language model, tuned it to the specific types of information they wanted to extract from these documents, and created a workflow that was much, much more compact. And we were able to sit it right next to the systems that, uh, you know, had the, had the document store coming in. So instead of sending traffic, you know, from one cloud to the next to process, we did it all locally. And with kind of a model that’s just way more cost-effective to run. And so I believe that we’re going to start to solve these problems with organizations and the industry is going to lean into not only workload placement, but thinking about the workflow itself, right? What’s the way to most optimize that particular outcome? And how do you leverage the right technologies to get in and do that?

Swapnil Bhartiya: And under what conditions does placing AI infrastructure closer to production applications reduce latency, data movement, and operational complexity?

Bryan Litchford: Yeah, when you think about a data store that you might have today, maybe it’s unstructured data, maybe it’s a database, that by itself is not generally like perfectly useful to an AI agent. Normally, if it’s a Let’s take a simplistic example of like a RAG application. You’d need some kind of vector database, you know, in between to help the language model understand what it’s looking at, right? Make it consumable ultimately to the LLM doing the work on the backend. And that workload would, if you went all in on a strategy of every single, AI agent I’m going to build will be hosted by a frontier model or maybe a hyperscaler, it means that all the data that you want that model to be up to date on either has to be fed periodically into that cloud of choice, has to be crunched down into the appropriate model, and then consumed by the LLM. And so you’re always moving things around unnecessarily, or you’re creating a really chatty application between, you know, call it an on-prem or near the core system vector database and leveraging, you know, an API to a hosted model somewhere else. And so I think enterprise is really concerned about security, latency, just not having those bits traverse any network, private or public. They’re going to start to look for those solutions to be closely coupled together. Right, and so that means the core system sits wherever the work is happening. The agentic system sits right next to it, potentially, you know, plugged into the same switching infrastructure, for example. That takes away your latency concern. It takes away your control concern because you are hosting that entire footprint end-to-end in one place for that particular workload. Now, again, I believe that we’re gonna be in a world where every workload might live in different locations over time. And that’s perfectly fine. I think there’ll be much more effective ways. And one of the things Rackspace has been heavily focused on is how do we make that enterprise AI consumption more manageable rather than just signing up for an API key and trying to control it from a finance-only view and a token scoreboard, if you will.

Swapnil Bhartiya: And what does running virtual machines and Kubernetes on the same platform change for teams who are managing existing applications along with, of course, newer workloads?

Bryan Litchford: Yeah, I think this is, you know, outside of AI, this is a great, you know, great question around how do the core platforms that maybe have been in organizations for years run alongside some of the newer, faster developed, you know, kind of modernized platforms that are more likely to be containerized? This debate, I think, has driven a lot of cloud selection over the years, right? In that everything was going to go to a container-based or container-first methodology. And so we’re going to ship that off to a hyperscaler. What we found is customers don’t necessarily want to just lift and shift anymore, right? They want to be able to run their core workloads. Maybe it’s a VMware-based workload, or sorry, VM-based workload, or a bare metal-based workload. Adjacent to their kind of newer applications, right? Rackspace has, you know, really selected and narrowed in on VMware as our core platform. And what we’re really excited about with VCF 9 is the ease at which both the VMs themselves as well as a managed Kubernetes environment can run on the same hypervisor infrastructure. This reduces the amount of kind of, I call it hardware bloat, that maybe in the past teams have had to deal with, where they maybe have a Kubernetes environment and a virtualization environment. They’re separate to each other, but it’s really a better use of a platform. And we’re leaning into how do we deliver That managed Kubernetes service and those VMs on a, you know, centralized single control plane so that for organizations working with these workloads, maybe on-prem in a core Rackspace data center or, you know, close to where the work is happening, we can give a very consistent experience depending on where that workload is, but the flexibility for our customers to actually move workloads from one location to the next, if that makes sense.

Swapnil Bhartiya: Now, for a customer evaluating Rackspace Cloud, where does VMware Cloud Foundation end and Rackspace’s operational responsibility begins?

Bryan Litchford: Yeah, fantastic one. You know, Rackspace, I like to say, is responsible for the dirt under the data center all the way up to the packet leaving our network and traversing out to an end user or out to the public internet. VMware Cloud Foundation 9 is just a core component of that overall service that we provide to customers, right? Rackspace ourselves, almost 30 years operating data centers, private cloud footprints, managing hyperscaler footprints, and more recently over the last 5 to 8 years really leaning into helping organizations adopt hybrid cloud, we have found that the VMware platform is just the absolute leader in the capability and maturity I think that’s required for a lot of these enterprise workloads that we support. Not just compatibility to various software platforms, but also, you know, tight integration from a hardware perspective. And so Rackspace’s responsibility is Not only to bring the VCF platform to our customers, but also to manage everything from the hardware procurement and supply chain challenges to the warranty and replacement of that equipment, to managing any failures, replacing that, handling the data center itself. Data center space and power is incredibly challenging to get your hands on right now. And so we’re a great partner to help navigate that space. And then We bring our 30 years of engineering and support experience to bear on our platforms. So no customer is, you know, effectively going it alone when they’re building out their platform. And from that, we get to provide tons and tons of input, experience, lessons learned into the design of each customer’s footprint going forward. And I think that resiliency and that scale is incredible.

Swapnil Bhartiya: Before we wrap this up, last question is more or less like about before moving production workload to Rackspace Cloud, what should a customer test to validate the expected performance and cost?

Bryan Litchford: We’ve been talking this whole time really around, you know, what’s the right place for workloads to live and how do you evaluate that? I think before, You make a decision on what to test. I think it’s really important to understand what you’re moving and why, right? Are you moving an application that has tech debt associated with it, or are you moving an application for better resilience, better performance? You’re looking for, you know, a more modern landing zone where that workload can continue to develop and grow and give you new capabilities. And then it’s really important to start thinking about what does this workload interact with. Is this a run once a month to crunch a bunch of numbers, you know, in report generation workload? Is this something that’s operationally critical to our day-to-day business? And so I need to design not just for footprint, you know, inside of Rackspace, but also have Rackspace build the disaster recovery and the cyber resiliency services around it. Thinking about how that workload’s going to live in, in a new location, I think, becomes really important to understand what you’re testing for. We have customers today that use Rackspace as a, you know, a dev environment, a playground, right? They don’t need to test for resiliency. They don’t need to to, you know, work with us on designs for, you know, how, how much overhead is built into their environment, how much can they scale up and scale down. But for other customers where they’re, you know, focused in on a mission-critical or what I would call a core application, it’s really important for them to, you know, talk to us about what are the expectations for this application. And I don’t want to know what the, you know, 95th percentile of performance looks like. I want to know what the 100th percent, the 99th percentile, if you will. I want to know the absolute peaks, the valleys. I want to understand the concerns around location, ensuring that, you know, there’s not too many workloads in, in one location, or disaster recovery is, is handled and accounted for. And it gives organizations, I think, a more robust checklist, if you will, to go through and check through. You know, from a platform perspective, you know, we’re using best-in-class hardware. We’re in some of the best— I like to call it our beachfront property from a data center perspective. We have fantastic interconnectivity, so latency to other services, whether a hyperscaler, a different cloud provider, We have that, you know, in spades and help customers test that all the time. But I think more important is understanding what’s that workload that you’re moving and why and building your test plan around that.

Swapnil Bhartiya: Bryan, thank you so much for walking us through these decisions. And of course, those who are watching, please go and check and learn more about Rackspace Technology at rackspace.com. And Bryan, thank you for your time. And I look forward to another great discussion with you. Thank you.

Bryan Litchford: Yeah, awesome. Thank you so much. Have a good one. Thanks, Adam.

Why AI Data Movement Changes Cloud Architecture | Danielle Cook, Akamai | TFiR

Previous article