Organizations can have multiple Airflow instances running on separate hardware, each with its own database, and sometimes in different geographical regions. How can we monitor these instances and get updates on their state without polling their databases or APIs? Is there a way to coordinate Dags from different Airflow instances?
This session answers the above questions by presenting a newly released plugin in the Apache Kafka Provider, which publishes a message to a configured Kafka topic for every Dag run or task instance state change event.
It will cover:
- what the plugin does
- what the published messages look like
- how to enable and configure it
- how to consume the messages
- some use cases
Christos Bisias
G-Research, Open Source Software Engineer