Manage Dags at scale (Dags versioning & package management)

This talk is all about how we at Jagex manage DAGs at scale. Focusing of following challenges faced and how we resolved them;

1. Keeping track of airflow state

2. Keeping track of each DAG state

3. DAGs as git submodules

4. Updating airflow with new dags

5. How to seamlessly automate airflow deployment

6. How to avoid package dependency conflicts