Zero‑Code Airflow at Scale: Building Enterprise Pipelines from CSV to Production DAGs

As Airflow adoption expands across large enterprises, a core challenge emerges: How to enable multiple teams to design and operate data pipelines without relying heavily on specialized engineering expertise. In this session, we will present a zero‑code, metadata‑driven Airflow framework built and deployed within a large financial services organization to accelerate pipeline development and onboarding at scale.

This framework allows users to define workflows using simple CSV or Excel inputs, which are automatically converted into YAML configurations and deployed as fully production‑ready Airflow DAGs using standardized templates on Astronomer. By leveraging a remote execution model and reusable DAG patterns, the solution supports orchestration across heterogeneous systems—including data warehouses, ingestion pipelines, and data quality frameworks—while maintaining enterprise‑grade governance, consistency, and observability.

The talk will walk through the high-level architecture, including Excel‑to‑YAML transformation logic, dynamic DAG generation patterns, and controls that enable non‑developer and cross‑functional teams to safely create and manage pipelines with minimal coding. We will also share lessons learned from taking this framework from initial design to enterprise‑wide production rollout, highlighting how it reduced onboarding time, enforced standardization, and scaled orchestration across teams.

Attendees will gain practical insights into implementing low‑code and no‑code orchestration on top of Apache Airflow, along with key architectural considerations for operating Airflow at enterprise scale.