Airflow and Data Mesh: Architecture, setup & monitoring

Ever thought how Airflow could play a pivotal role in data mesh architecture, hosting thousands of DAGs and hundreds of thousands daily running tasks, let’s find out!

In this workshop we will cover:

• What requirements should be met to adopt a data mesh architecture.
• Airflow and it’s role in Data Mesh
• How to set up monitoring for running scalable Airflow
• How to manage cross team DAG dependencies
• How to manage multi-tenancy
• Challenges and factors to keep in mind to make it a success

This workshop should be suitable for any Architect, Devops and Data Engineers. At the end of this workshop you would have solid understanding for using Airflow as pivotal part in an organisation’s data mesh framework.