What happens when your Airflow footprint grows from a single team’s scheduler to a fully managed enterprise data platform? You stop thinking about individual DAGs and start thinking about fleet operations, tenant isolation, automated upgrades, and self-service authoring at scale.
In this talk, we share our journey building a multi-tenant orchestration platform on Apache Airflow running hundreds of isolated instances on Kubernetes. We will dive deep into:
- The Architecture: How we achieved strict tenant isolation and automated infrastructure provisioning on Kubernetes.
- Fleet Operations: Our strategy for handling seamless, zero-downtime upgrades across hundreds of instances.
- The Abstraction Layer: How we built a self-service experience that allows any team—technical or non-technical—to go from idea to production pipeline without managing infrastructure.
Whether you are looking to scale your current Airflow deployment or are managing Airflow at an enterprise level, you will walk away with actionable architectural patterns and real-world operational lessons.
Pavan Kumar Naikal
Data Engineer