There are many Airflow tutorials. However, many don’t show the full process of sourcing, transforming, testing, alerting, documenting, and finally supplying data. This talk with go over how to piece together an end-to-end Airflow project that transforms raw data to be consumable by the business. It will include how various technologies can all be orchestrated by Airflow to satisfy the needs of analysts, engineers, and business stakeholders.
The talk will be divided into the following sections:
Introduction: Introducing the business problem and how we came up with the solution design
Data sourcing: Fetching and storing API data using basic operators and hooks
Transformation and Testing: How to use dbt to build and test models based on the raw data
Alerting: Alerting the necessary parties when any part of this DAG fails using Slack
Consumption: How to make dynamic data accessible to business stakeholders