The talk will cover how we use Airflow at the heart of our Workflow Management Platform(WFM) at Booking.com, enabling our internal users to orchestrate big data workflows on Booking Data Exchange(BDX).
High level overview of the talk:
Adapting open source Airflow helm chart to spin up Airflow installation in Booking Kubernetes Service (BKS)
Coming up with Workflow definition format (yaml)
Conversion of workflow.yaml to workflow.py DAGs
Usage of Deferrable operators to provide standard step templates to users
Workspaces (collection of workflows), using it to ensure role based access to DAG permissions for users
Using okta for authentication
Alerting, monitoring, logging
Plans to shift to Astronomer
