Migrating Airflow-based Spark jobs to Kubernetes - the native way

Speaker(s): Roi Teveth Itai Yaffe
When: (Jul-8 06:00 UTC)

At Nielsen Identity Engine, we use Spark to process 10’s of TBs of data. Our ETLs, orchestrated by Airflow, spin-up AWS EMR clusters with thousands of nodes per day.

In this talk, we’ll guide you through migrating Spark workloads to Kubernetes with minimal changes to Airflow DAGs, using the open-sourced GCP Spark-on-K8s operator and the native integration we recently contributed to the Airflow project.