Multi-Instance Asset Synchronization - push or pull ?

As Data Engineers, our jobs regularly include scheduling or scaling workflows.

But have you ever asked yourself, can I scale my scheduling ?

It turns out that you can! But doing so raises a number of issues that need to be addressed.

In this talk we’ll be:

• Recapping Asset-aware scheduling in Apache Airflow
• Discussing diverse methods to upscale our scheduling
• Solving the issue of maintaining our Airflow Asset synchronized between instances
• Comparing our professional push based solution and the built-in solution from AIP-82 and the pros and cons of each method.

I hope you will enjoy it!