Airflow as the next gen of workflow system at Pinterest

At Pinterest, our current workflow system, called pinball, has served the data pipeline orchestration demands well for years. However, with the rapid increasing execution demand the system started to expose scalability and performance issues. Therefore we decided to look for a new solution to better address the issues and serve the workflow scheduling demand, and we chose Airflow as our next generation of workflow. In this talk we discuss how we made the decision to on board to Apache Airflow, and beyond the out-of-box features and experience what improvements we made to better support the business need at Pinterest.