One Codebase, Many Distributions: Airflow’s Modular Approach

A harness humans and agents can both follow

Airflow’s evolution toward a client-server architecture faced a fundamental challenge: splitting a monolithic codebase into independent distributions (airflow-core, task-sdk, providers) without triggering dependency hell. Traditional PyPi packaging and code duplication both fail at Airflow’s scale.

Airflow 3.2 solves this through modular isolation and shared libraries using in-repository symlinks. This approach ensures each distribution ships with the exact version of shared code it requires, eliminating runtime version conflicts and allowing for independent dependency management. We have already migrated 10+ critical components—including the config parser, observability, and secrets masking—into this shared model.

Independent upgrade paths: ship a provider fix without touching your scheduler — and see honestly how far core/task-sdk decoupling actually is today.

Maintenance: security fixes flow across different components on their own release cadence, not airflow core’s.

API-first workers (AIP-72): lightweight execution without direct database access.

An unexpected result: the same machinery that made the monorepo navigable for humans — uv workspaces, 170 prek hooks, enforced module boundaries — turned out to be exactly what coding agents need. Agent-assisted contributions now arrive pre-validated against the full harness. That’s a genuine gain in reviewability, and a genuine flood. We’ll talk about both, and about how modularity may become our main tool for steering contributions away from the parts of Airflow that still run on institutional knowledge.