Airflow has been used by many companies as a core part of their internal data platform. Would you be interested in finding out how Airflow could play a pivotal role in achieving data engineering excellence and efficiency using modern data architecture. The best practices, tools and setup to achieve a stable but yet cost effective way of running small or big workloads, let’s find out!
In this workshop we will review how an organisation can setup data platform architecture around Airflow and necessary requirements.
- Airflow and it’s role in Data Platform
- Different ways to organise airflow environment enabling scalability and stability
- Useful open source libraries and custom plugins allowing efficiency
- How to manage multi-tenancy, cost savings
- Challenges and factors to keep in mind using Success Matrix!
This workshop should be suitable for any Architect, Data Engineer or Devops aiming to build/enhance their internal Data Platform. At the end of this workshop you would have solid understanding of initial setup and ways to optimise further getting most out of the tool for your own organisation.
M. Waqas Shahid
Delivery Hero, Principal Data Platform Engineer