Speaker(s):
Have you faced a scenario where 100 worker slots are available to run the Tasks, but you have 100 DAGs waiting on a Sensor that’s currently running but idle, waiting for something to happen? Ultimately, you got frustrated as you could not run anything else - even though your entire Airflow cluster was essentially idle. This is exactly where the concept of Deferrable Operators is very useful.
This talk aims to give a brief introduction to solving this problem using Deferrable or Async Operators and how to implement it for your use case.
This talk also aims to explain:
- Introduction to deferrable operator
- Introduction to Triggers
- Using deferrable operator - real-world use case
- Writing deferrable operators
- Triggering deferral
- Writing Triggers
- Example of deferrable S3 Operator
- Advantages over Smart Sensors and reschedule mode for sensor
- Guide to implement your deferrable operator with astronomer-providers repository as an example to start with
- References for useful concepts and libraries: Concurrency, Python asyncio