The Why and How of Running a Self-Managed Airflow on Kubernetes

Speaker(s):
Parnab Basak
Parnab Basak
By Parnab Basak

  Download slides

Today, all major cloud service providers and 3rd party providers include Apache Airflow as a managed service offering in their portfolios. While these cloud based solutions help with the undifferentiated heavy lifting of environment management, some data teams are also looking to operate self-managed Airflow instances to satisfy specific differentiated capabilities. In this session, we would talk about:

  1. Why should you might need to run self managed Airflow

  2. The available deployment options (with emphasis on Airflow on Kubernetes)

  3. How to deploy Airflow on Kubernetes using automation (Helm Charts & Terraform)

  4. Developer experience (sync DAGs using automation)

  5. Operator experience (Observability)

  6. Owned responsibilities and Tradeoffs

A thorough understanding would help you understand the end-to-end perspectives of operating a highly available and scalable self managed Airflow environment to meet your ever growing workflow needs.