IBM releases CodeFlare for Big Data and AI Workflow integration

CodeFlare is all about end-to-end workflows and pipelines and aims to drastically reduce the time it takes to set up, run, and scale machine-learning tests. The motivation behind CodeFlare, according to Priya Nagpurkar, Director of Cloud Platform Research at IBM Research, “was the emergence of these converged workflows. So you have AI, machine learning, big data, and even simulations and modeling, all coming together into tightly integrated workflows.” But how does this differ from traditional AI/ML platforms? According to Nagpurkar, the difference is, “When I can think about my logic, and I have higher-level interfaces, and I don’t have to worry about the runtime aspects, how do I scale? How do I map it to massive infrastructure?” In the end, CodeFlare deals with workflows as a whole, instead of individual elements.

But how scalable is CodeFlare? According to Nagpurkar, “We are really pushing together with these AI/ML communities and Kubernetes, to go to thousands of cores and in the future from hundreds of thousands. And I think the second important aspect of scalability is also data. I spoke about compute, but we also want to be able to scale to bigger and bigger data sets, and run those on the cloud. So you can imagine a single CodeFlare pipeline being able to deal with petabytes of data.”

IBM decided to open-source CodeFlare in part because they’d been a major player in the open-source community for a long time. And with their acquisition of Red Hat, they’re embracing it even more. “We think that open source is the way to accelerate innovation that’s required in this space,” quips Nagpurkar.

She also believes that ML is just as critical as digital transformation and cloud. To this point, she says, “A key element of where innovation is happening, is data-driven, real-time insights to help serve customers better and add automation.” Nagpurkar adds, “We want the cloud to seem like a seamless extension all the way out to the edge. And we want those properties to scale as needed.”

Video Summary was written by Jack Wallen