Airflow at Zoox: A journey to orchestrate heterogeneous workflows

The workflow orchestration team at Zoox aims to build a solution for orchestrating heterogeneous workflows encompassing data, ML, and QA pipelines. We have encountered two primary challenges: first, the steep learning curve for new Airflow users and the need for a user-friendly yet scalable development process; second, integrating and migrating existing pipelines with established solutions.

This presentation will detail our approach, as a small team at Zoox, to address these challenges. The discussion will cover the scope and scale of Airflow within Zoox, including current applications and future directions. Furthermore, we will share our strategies for simplifying the Airflow DAG creation process and enhancing user experience. Finally, we will present a case study illustrating the onboarding of a heterogeneous workflow across Databricks, AWS, and a Zoox in-house platform to manage both on-prem and cloud services.