Self-Service DAGs: Event-Driven Design for GitHub Actions and Airflow at Lyft

At Lyft, driver pay configs on GitHub must be validated through Airflow DAGs before merging. However, Scientists and Analysts who change configs are not familiar with Airflow. How do we make such validation self-service while meeting SOX compliance?

This talk presents a design pattern for bidirectional GitHub-Airflow integration: GitHub Actions trigger DAGs, and DAGs push results back as PR status checks via the GitHub Commit Status API. We cover event-driven push-style vs traditional polling style, and why an event-driven push-style works well with Dynamic Task Mapping. This pattern aligns with Airflow 3’s event-driven scheduling vision. We also discuss how SOX requirements shaped this design.

GitHub-Airflow integration is not just hitting APIs. We also address real-world challenges: choosing between GitHub API and GitPython for read/write patterns, preventing DAGs from overriding built-in PR checks, blocking merges during long-running execution, and handling PR rebases that orphan commit statuses. Our solution uses a persistent mapping table to propagate results across rebased commits.

Attendees will leave with a reusable blueprint for event-driven GitHub-Airflow integration.