Program

At Airflow Summit 2025 we showed how we generate data quality DAGs from YAML using DAGFactory, putting check authoring in the hands of 30+ engineers, analysts, and BI users on Cloud Composer. It worked and it created a new problem. Check coverage grew faster than our ability to respond to it, and on-call was spending a median of one to two hours per alert just deciding whether it was real.

Almost none of that time was judgment. It was gathering: querying historical values for the failing metric, checking upstream task state, looking for schema drift, correlating recent deploys, comparing sibling checks on the same table. All of it was context Airflow already had.

This talk covers how we turned that gathering loop into a triage DAG. A failing check triggers parallel context-assembly tasks, hands the result to an LLM constrained to a strict output schema, and posts a structured verdict to Slack: true positive, false positive, or inconclusive, with confidence, reasoning, and cited evidence. The engineer approves a rerun, suppresses, or escalates with one click - Airflow stays in control of every action taken. Median time to a triage decision dropped from one to two hours to under ten minutes.

We’ll walk through the DAG design, the output schema, how the Slack approval loop doubles as a labeled evaluation set, and what we measure to know the model isn’t quietly wrong, including false-negative rate and how often engineers override the verdict. We’ll also cover what broke along the way: hallucinated root causes, context bloat, cost per triage, and the open question of whether humans keep reviewing carefully once the verdicts start getting good.

Agentic AI systems are difficult to ship safely: their behavior is non-deterministic, traditional assertions only cover part of the problem, and manual evaluation quickly becomes a release bottleneck.

In this talk, I’ll share how we built Pegasus, an automated evaluation platform for agentic applications, and used Apache Airflow as its orchestration backbone. The platform combines deterministic tests with LLM-as-a-Judge and Agents-as-Judges evaluations. Airflow coordinates the evaluation harness across scheduled and on-demand runs. a DAG checks the commit currently deployed across different environments, determines whether that version has already been evaluated, and runs the required test suites only when needed. The same workflow can also be triggered asynchronously during development and release processes.

This approach automated roughly 80% of tests that previously required manual execution and helped move our production release cadence from about once a month to twice or more a week.

Attendees will leave with a practical architecture for orchestrating AI evaluations with Airflow, patterns for combining scheduled and on-demand evaluation workflows, and lessons for making agentic AI testing repeatable, observable, and useful as part of a production release process.

Robot foundation model training is data-bound, not compute-bound. GPUs sit idle while heterogeneous trajectory data, synchronized camera streams, IMU, joint-state, and force/torque signals, waits to be transformed into training-ready tensors. Production teams generate tens of terabytes daily, and generic data-loader patterns don’t compose the way this data needs.

This talk shares a compositional DAG architecture built on Airflow for exactly this problem. Each ingestion step declares its own CPU/GPU/memory profile, ships as a versioned container image, fans out across Kubernetes workers using a distribute-wait pattern, and is selectively gated to skip re-execution of unaffected steps when downstream logic changes. Business logic runs off-orchestrator so the scheduler’s worker pool never saturates under compute load.

In production, this eliminated over 30% of training slowdown while coordinating an 88 TB cache across 16 H100 nodes.

Attendees will leave with concrete DAG design patterns for per-step resource specification, selective re-execution, and orchestrator/compute separation, applicable to any team running heterogeneous, resource-heavy pipelines on Airflow, not just robotics.

In enterprise SaaS, orchestrating ML models on sensitive workforce data requires zero compromise on data privacy and multi-tenant isolation. Workday’s Scheduling & Labor Optimization product uses Airflow to power predictive workforce forecasts across thousands of customer tenants—where each customer strictly owns their data and model artifacts. This session demonstrates how Workday leverages Airflow for multi-tenant ML pipeline orchestration. We cover how Airflow dynamically materializes isolated ML DAGs per tenant, enforces strict execution boundaries via TENANT_KEY metadata, and automates lifecycle operations for customer opt-in and opt-out workflows, including mandatory 14-day data/model purges.
Key Problems Solved:

1. Strict Data & Model Privacy: Isolating tenant data and training models exclusively on tenant-owned datasets.
2. Lifecycle Compliance: Triggering DAG creation on opt-in and managing grace periods with enforced 14-day purges on opt-out. 3. Operational Efficiency: Scaling thousands of dynamic DAGs using VPA/Scaleops autoscaling without scheduler overhead.

Session Outline :

1. Overview & Challenge
2. Multi-tenant ML privacy in SaaS.
3. Dynamic DAG Architecture
4. Event-driven materialization and TENANT_KEY isolation.
5. Lifecycle Compliance
6. Dynamic enrollment and automated 14-day deletion pipelines. Scaling & Infrastructure
7. Resource optimization with ScaleOps/VPA

Takeaways:

1. Design patterns for dynamic per-tenant DAG generation.
2. Enforcing tenant isolation, security boundaries, and model governance in Airflow.
3. Building compliance workflows for dynamic opt-in/opt-out retention policies.

Notion runs in four AWS regions today, with more on the way, and every data pipeline has to respect where a customer’s data lives. Airflow on Astronomer is the control plane that holds this together: two Astro deployments, 500+ DAGs, and region-aware routing to EMR, EMR Serverless, Ray on Anyscale, Kafka, Snowflake, Databricks, and our vector database.

This talk covers the platform patterns that let a small team operate all of that: a generated cell and region model shared by every DAG, so new regions and retired cells propagate without touching pipeline code; DAG factories that fan one config out per region; and per-environment, per-region compute isolation, including the custom Anyscale operator we built and the cloud migrations we ran from DAG config alone.

We then zoom into one concrete workload: indexing Jira, Slack, Google Drive, GitHub, and a dozen other connectors for Notion AI. Kafka feeds Spark and Ray embedding jobs that write to region-local vector indexes. This workload shows both where our abstractions held up and where they broke down, including the per-region DAG file explosion we are still paying down.

Attendees will leave with practical patterns for designing region-aware Airflow platforms: how to model regions and cells, generate DAGs without duplicating business logic, isolate compute by environment and geography, and evolve infrastructure as regions are added, migrated, or retired.

At last year’s Airflow Summit, we shared how we built a multi-cluster orchestration layer on top of Apache Airflow to run ML workloads across multiple Kubernetes GPU clusters.

Once hundreds of ML engineers started running GPU pipelines in production, we discovered that orchestration alone is not enough. Operating multi-cluster GPU infrastructure introduces new challenges: controlling GPU allocation across teams, observing pipelines across clusters, and helping users run workloads efficiently without wasting expensive GPU resources.

In this talk, we’ll show how our Airflow platform evolved from a workflow orchestrator into an operational control plane for GPU infrastructure. We’ll cover custom scheduling strategies that dynamically route workloads across clusters using Airflow policies and resource awareness, integration with HAMI to improve GPU utilization, and AIOps workflows with KeepHQ that detect underutilized CPU, RAM, and GPU resources. We’ll also present powerful dashboards and AI-assisted tools that reduce Time2Market and simplify debugging while keeping infrastructure complexity hidden.

We’ll be happy to share how our platform continues to evolve with Apache Airflow.