Netflix has open sourced its Maestro workflow orchestrator, which is designed to manage large-scale Data/ML workflows such as data pipelines and machine learning model training pipelines. Netflix Maestro provides a wide range of features tailored to the varied requirements of both engineers and non-engineers. It incorporates common functionalities and reusable patterns that can be applied to different scenarios in a flexible and loosely coupled manner.

Over the past year, Netflix Maestro has experienced an impressive 87.5% growth in executed jobs. Currently, Maestro initiates thousands of workflow instances and handles an average of half a million jobs each day, reaching up to 2 million jobs on exceptionally busy days.

Maestro is a fully managed workflow orchestrator that provides Workflow-as-a-Service to thousands of end users, applications, and services at Netflix. It supports a wide range of workflow use cases, including ETL pipelines, ML workflows, AB test pipelines, pipelines to move data between different storages, etc.

Maestro’s horizontal scalability allows it to efficiently handle numerous workflows as well as multiple jobs within each workflow.

Click here to find the Maestro code repository.