At Coinbase, Airflow is adopted by a wide range of applications, and used by nearly all the engineering and data science teams. In this session, we will share our journey in improving the productivity of Airflow users at Coinbase. The presentation will focus on three main topics:
-
Monorepo based architecture: our approach of using a monorepo to simplify DAG development and enable developers from across the company to work more efficiently and collaboratively.
-
Tailored testing environment: our tailored Airflow testing environments that cater to users of different profiles, helping them to test their code more efficiently and with greater confidence.
-
AirAgent: our in-house solution for Airflow continuous deployment, which puts Airflow deployment in self-driving mode and supports deploying any code changes related to Airflow (DAGs, plugins, configurations, dependency changes, etc.) without downtime.