Autoscaling in Airflow - Lessons learned

(07:00-07:50 UTC) -
Add to Calendar 07/16/2021 7:00 AM 07/16/2021 7:50 AM UTC Airflow Summit: Autoscaling in Airflow - Lessons learned

Autoscaling in Airflow - what we learnt based on Cloud Composer case. We would like to present how we approach the autoscaling problem for Airflow running in Kubernetes in Cloud Composer: how we calculate our autoscaling metric, what problem we had for scaling down and how did we solve it. Also we share an ideas on what and how we could improve the current solution

https://airflowsummit.org/live

Autoscaling in Airflow - what we learnt based on Cloud Composer case. We would like to present how we approach the autoscaling problem for Airflow running in Kubernetes in Cloud Composer: how we calculate our autoscaling metric, what problem we had for scaling down and how did we solve it. Also we share an ideas on what and how we could improve the current solution