In large organizations, multiple Apache Airflow instances often arise organically—driven by team-specific needs, distinct use cases, or tiered workloads. This fragmentation introduces complexity, operational overhead, and higher infrastructure costs. To address these challenges, we developed the “Orchestration Frederator,” a solution designed to unify and horizontally scale multiple Airflow deployments seamlessly.
This session will detail our journey in implementing Orchestration Frederator, highlighting how we achieved:
-
Horizontal Scalability: Seamlessly scaling Airflow across multiple instances without operational overhead.
-
End-to-End Data Lineage: Constructing comprehensive data lineage across disparate Airflow deployments to simplify monitoring and debugging.
-
Multi-Region Support: Introducing multi-region capabilities, enhancing reliability and disaster recovery.
-
Unified Ecosystem: Consolidating previously fragmented Airflow environments into a cohesive orchestration platform.
Join us to explore practical strategies, technical challenges, lessons learned, and best practices for enhancing scalability, reliability, and maintainability in large-scale Airflow deployments.
Chirag Todarka
Software Engineer at Doordash
Alvin Zhang
Software Engineer at Doordash