Author: Arnaud Alcabas, Solutions Architect, Scaleway
Bio: Arnaud Alcabas has been a Solutions Architect at Scaleway for 2 years. Previously a Lead Technical Consultant after being a Project Manager in a Ops division, he spent 15 years at the heart of the infrastructure of many clients to support them in automating their infrastructure and application deployments to ensure reproducibility, security, and agility, either on baremetal, virtual machines or container platform.

High Performance Computing (HPC) comes with a complete set of new challenges compared to more traditional computing.

Leveraging parallel data processing needs an efficient way to split data between processes, scheduled on a large number of computing nodes, and then gather the output without corrupting the end result. It may not be the only type of parallelism needed for AI/ML, but it is still front and center in most use-cases.

Fast networking and distributed file systems with a guarantee of coherence are among those challenges. Technologies like Infiniband and Lustre are often used to tackle these needs.

But scheduling the computing jobs optimally is also key to HPC clusters.

Scheduling is in itself a very interesting topic in computer science, with various algorithms available to ensure maximum efficiency without triggering resource starvation. And HPC and Kubernetes scheduling are inherently different.

HPC scheduling

Without a scheduler, an HPC cluster is just a bunch of servers, each doing their own share of work without collaborating.

There are multiple scheduling engines available, developed in universities and companies for their specific needs, sometimes then patented or distributed in an open-source version.

But a popular one is SLURM (Simple Linux Utility for Resource Management).

Open-sourced under GNU GPL, it is used in multiple HPC clusters such as the Jean Zay Supercomputer due to several key advantages:

• ability to partition compute resources and allow giving exclusive access to resources
• monitoring and managing job execution across multiple users and resources to allow canceling jobs and sharing resources more fairly
• queuing jobs to allow an execution of the associated workload in orderly manners without losing compute time.
• modular thanks to a plugin system, SLURM allow adding features such logging, security and authentication

All of these functions make SLURM still relevant in HPC clusters today and knowing how to use it allows users to execute their workload on several different clusters.

But SLURM is not without drawbacks.

Firstly, even with plugins leveraging GRES (Generic RESources), handling GPU access, CUDA configuration, sharding and MIG is still difficult making AI/ML more complex on SLURM.

Secondly, it is specifically designed for long standing jobs and leveraging SLURM for any other use-case is not practical, leaving compute resources idle if no computing jobs are available. This also leaves out other data pipeline workflows typically more suited to other deployment platforms like Kubernetes.Training is fine on SLURM, inference is not.

To schedule service workloads, something else must be used.

Kubernetes Scheduling

Kubernetes is the De Facto platform for microservices and applications deployment with one of the greatest desirability of 2023.

Whereas SLURM support for containers is limited (container images must already be present on the nodes, no container network, rootless execution), Kubernetes is a full fledge container orchestrator, able to leverage multiple container runtimes to execute a variety of workloads such replicated deployments, jobs and stateful services.

The default Kubernetes scheduler, kube-scheduler, focuses on finding the best target node for each pod leveraging topology aware mechanism, node affinity, resource allocation and optimization. But it works by evaluating each pod against all available nodes, which can take time on large clusters. Since it needs to filter nodes vs multiple criteria to fit numerous workload types, it is not the best choice when needing to schedule and queue jobs.

Nevertheless, Kubernetes’ always growing adoption and availability as a managed service in most cloud providers make it an ideal target for must use cases, with most developers already knowing how to use it, contrary to SLURM for newcomers to the AI/ML or HPC world.

This leads to a need to converge these two worlds in an efficient way: having the best of HPC scheduling on top of Kubernetes.

The best of both worlds?

Running both Kubernetes and native SLURM on the same compute nodes is out of the equation since both would compete for resources without knowing the constraints and expectations of the other.

But CoreWeave SUNK (SlUrm oN Kubernetes) announced in a blog post for Kubecon 2023, scheduled to be open-sourced in 2024, would be a fitting candidate to make this into reality, leveraging Custom Resource Definition and SLURM pods to make someone used to SLURM right at home on a Kubernetes cluster, also leaving more traditional workloads live in the same compute space.

While this approach is very exciting, another candidate taking another direction is Apache Yunikorn. Initially developed by Cloudera and open-sourced in 2019, it can be used as drop-in replacement for kube-scheduler or as a separate scheduler and can handle the requirement of batch jobs and long standing services altogether.

This feels more cloud native and compatible with cloud provider Managed Kubernetes Services, since it uses a mechanism like Gang scheduling to pro-actively request resources with placeholder pods, which in turns help the cluster-autoscaler do its job.

It does not provide SLURM familiar interface to HPC enthusiasts, but it is compatible with AI/ML and HPC workloads, while allowing cluster owners the flexibility to have both model training and inference done in the same place, using the same resources without needing to switch stack.

With an Helm chart available and version 1.4.0 already out, it was up and running on a Scaleway Kapsule cluster under a minute, ready to create partitions and queues to schedule jobs on GPU instances using traditional Kubernetes workloads in a HPC way.


$ time helm install yunikorn yunikorn --version 1.4.0 \
--repo https://apache.github.io/yunikorn-release \
--create-namespace --namespace yunikorn \
--set enableSchedulerPlugin=true --wait

real    0m34.449s

With this aspect of the problem out of the way, AI/ML on Managed Kubernetes looks even easier than before, especially if you pair it with a great project like Kubeflow to tailor your Kubernetes experience to AI/ML with pipelines and notebooks, model training and serving.

To learn more about Kubernetes and the cloud native ecosystem, join us at KubeCon + CloudNativeCon Europe in Paris from March 19-22.