Today, all major cloud service providers and 3rd party providers include Apache Airflow as a managed service offering in their portfolios. While these cloud based solutions help with the undifferentiated heavy lifting of environment management, some data teams are also looking to operate self-managed Airflow instances to satisfy specific differentiated capabilities. In this session, we would talk about:
-
Why should you might need to run self managed Airflow
-
The available deployment options (with emphasis on Airflow on Kubernetes)
-
How to deploy Airflow on Kubernetes using automation (Helm Charts & Terraform)
-
Developer experience (sync DAGs using automation)
-
Operator experience (Observability)
-
Owned responsibilities and Tradeoffs
A thorough understanding would help you understand the end-to-end perspectives of operating a highly available and scalable self managed Airflow environment to meet your ever growing workflow needs.