Speaker(s):
Sep-10 17:10-17:35 in California East
Add to Calendar 09/10/2024 5:10 PM 09/10/2024 5:35 PM America/Los_Angeles AS24: DAG Dependency Management across Lowes

DAG dependency is already a solved use case for the same Airflow instance. But what happens when you have 50+ Airflow instances across teams and the workflow of one or many depends on others? By leveraging sensors and datasets we have created a custom operator that brings in the capability of cross-cluster dependencies. It works with our OnPrem Kubernetes architecture which is responsible for deployment of the custom operators throughout the entire Organization.

California East
By

DAG dependency is already a solved use case for the same Airflow instance. But what happens when you have 50+ Airflow instances across teams and the workflow of one or many depends on others? By leveraging sensors and datasets we have created a custom operator that brings in the capability of cross-cluster dependencies. It works with our OnPrem Kubernetes architecture which is responsible for deployment of the custom operators throughout the entire Organization.