Airflow at Shopify: Keeping Users Happy while Running Airflow at Scale

Two years after starting our Airflow adoption, we’re running over 10,000 DAGs in production. On this journey we’ve learned a lot about Airflow management and stewardship and developed some unique tools to help us scale.

We’re excited to share our experience and some of the lessons we’ve picked up along the way. In this talk we’ll cover:

  • The history of Airflow at Shopify
  • Our infrastructure and architecture
  • Custom tools and procedures we’ve adopted to keep Airflow running smoothly and our users happy