Join me at this year’s Airflow Summit as we delve into a pivotal evolution for Apache Airflow: The integration of data awareness.
Airflow has long excelled as a workflow orchestration tool, managing complex workflows with ease and efficiency. However, it has operated with limited insight into the data it manipulates or the assets it produces. This talk will explore the implications and benefits of embedding deeper insights about these outputs directly into Airflow.
We’ll start with a retrospective on Airflow’s origins and its task-centric approach, discussing why Airflow has thrived even without a focus on data awareness. We’ll then examine how enhancing the connection between tasks and the assets they produce can significantly boost Airflow’s utility and value for its users. Finally, we’ll consider new features that can be developed with this enhanced level of understanding, empowering data engineers with tools for more efficient, reliable and insightful operations.
