Airflow at GoDaddy: From on-prem to cloud to PaaS

Discover the transformation of Airflow at GoDaddy: from its initial deployment on-prem to its migration to the cloud, and finally to a Single Pane Orchestration Model.

This evolution has streamlined our Data Platform and improved governance. Our experience will be beneficial for anyone seeking to optimize their Airflow implementation and simplify their orchestration processes.

• History and Use-cases
• Design, Organization decisions, and Governance: Examining the decision-making process and governance structure.
• Migration to Cloud:Process of transitioning Airflow from on-premises to the cloud.
• Data Processing engines used with Airflow for Data Processing.
• Challenges: Obstacles faced during and after migration and how they were overcome. *Demonstrating how Airflow can be integrated with a central Glue Catalog and Data Lake Mesh model.
• Single Pane Orchestration (PAAS) and custom re-usable Github Actions: Examining benefits of using a Single Pane Orchestration model
• Monitoring