Building Blocks, not Factories: Abstractions that enable, not obscure

Many teams develop their own “Dag factory” to make Airflow easier to use in their organizations. This can help their users avoid python and configure Dags in a simpler manner. However, there is a huge spike in the difficulty curve of writing a DAG if it requires logic that does not fit into the confines of the Dag factory. If you want to create such a DAG, you are then having to completely abandon the pre-made framework and go back to writing a pure airflow DAG. Instead, I will present a different perspective that instead of producing entire DAGs, you should create pre-made task groups that can be dropped into a DAG to cover common steps, but in a manner that maintains a smooth difficulty curve if you want to add customer elements.