One Codebase, Many Distributions: Airflow’s Modular Approach

Airflow’s evolution toward a client-server architecture faced a fundamental challenge: splitting a monolithic codebase into independent distributions (airflow-core, task-sdk, providers) without triggering dependency hell. Traditional PyPi packaging and code duplication both fail at Airflow’s scale.

Airflow 3.2 solves this through modular isolation and shared libraries using in-repository symlinks. This approach ensures each distribution ships with the exact version of shared code it requires, eliminating runtime version conflicts and allowing for independent dependency management. We have already migrated 10+ critical components—including the config parser, observability, and secrets masking—into this shared model.

This architecture unlocks:

Zero Version Conflicts: Mix and match airflow-core and task-sdk versions seamlessly.

Streamlined Maintenance: Automatic security fixes across all components.

AIP-72 Vision: Lightweight workers with API-first communication, removing the need for direct database access.

Join us to explore how shared libraries transform Airflow’s monorepo into a modular, scalable, and truly distributed orchestration platform!