Workshop: Best Practices for running Airflow on Kubernetes

Speaker(s): Daniel Imberman
When: (Jul-10 16:00 UTC)
-----

Participation in this workshop requires previous registration and has limited capacity. Get your ticket at https://ti.to/airflowsummit/w2

With the growing popularity of Kubernetes in the tech industry, many Airflow users have started deploying Airflow on Kubernetes. With this migration, Airflow users have a lot of questions regarding best practices (Do I mount my DAGs via volume or bake them into the image? How do I inject my secrets? etc.).

To answer many of these questions, we invite you to join Daniel Imberman (Apache Airflow Committer and creator of the KubernetesExecutor) and Greg Neiheisel (Chief Architect of Astronomer.io) to learn all of the ins-and-outs of running airflow on Kubernetes.

Topics covered will include: The official Airflow Helm chart, Local development with KinD (Kubernetes-in-docker), Autoscaling with KEDA, KubernetesExecutor best-practices, and alerting/debugging airflow issues on Kubernetes.

Astronomer.io services over 300 airflow deployments across a wide range of Kubernetes services (including GKE, EKS, AKS, and IKS), and we are excited to bring this experience and expertise to you.

Prerrequisites:

  • The facilitators of the workshop will not provide a Kubernetes cluster for working. So, you need to have your own Kubernetes cluster or a local KinD cluster (and kubectl), as well as a Python environment (3.6 or later).