Program

This session explores the next phase of Dag versioning in Airflow and the practical questions users face in real deployments. Dag versioning moved Airflow beyond a “latest only” model, but it also introduced confusion around why Dag versions keep increasing, what disabling Dag bundle versioning actually does, what creates a new version, and how users should think about clears, reruns, and backfills after a Dag changes. I will examine a common misconception: disabling bundle versioning does not stop Dag version changes. I will also connect Dag versioning to Dag delivery in Airflow 3, showing how Git backed Dag bundles provide a more native alternative to git-sync in Helm-based deployments.

Migrating a production Airflow deployment from version 2 to 3 without disrupting hundreds of DAGs across multiple teams sounds scary (and it is). In this talk I will share how we migrated versions without a big-bang cutover, without weeks of cross-team change requests, and without leaving our pipelines in a broken state.

I’ll walk through how we built a compatibility layer to make sure our code runs on both versions during the migration, how we used AI-tooling to orchestrate 400+ DAG changes and how our on-demand ephemeral environments - full k8s deployments deployed for each pull request - helped us experiment and test all the required changes.

Most important of all, I will share what we learned, where we failed and what we would do better next time.

What if your Airflow DAG could orchestrate robots, thermal chambers, and silicon tests, not just code?

Silicon validation labs rely on scarce, stateful physical resources: robotic handlers, DUT boards, thermal/power systems, instruments, and shared hardware queues. Teams often coordinate these via spreadsheets and ad hoc reservations, causing contention, idle gaps, conflicts, poor observability, and slow triage.

This talk presents a closed-loop orchestration model where Apache Airflow is the control plane for a software-defined validation lab. A central DAG coordinates robotic handling, thermal/power setup, stress and performance runs, and parametric characterization on hosts connected to silicon. It continuously ingests hardware health, measurements, and test outcomes, then feeds results into AI-assisted analysis to choose the next physical action: refine parameters, schedule follow-up experiments, or trigger mitigation.

Using Edge workers on dedicated lab machines, we replace manual coordination with reliable, auditable orchestration. The same pattern extends beyond silicon to robotics labs, device farms, and other cyber-physical environments.

Debugging Airflow failures in production can be harder than building the pipelines themselves. Engineers often encounter issues such as disappearing DAGs, hanging tasks, missing logs, zombie tasks, or sudden performance degradation, often with little visibility into the root cause.

Over the past year, while supporting multiple Airflow deployments and integrations, we investigated several such incidents across different teams and environments. This session shares lessons from these real debugging cases and explains how the issues were diagnosed and resolved.

We will walk through incidents involving scheduler behaviour, concurrency limits, memory pressure, and process-level failures. For each case, we highlight the symptoms, the investigation approach, and the root cause.

Attendees will learn

• How to systematically debug complex Airflow failures
• Which components commonly hide the root cause
• Practical signals to watch in logs and metrics

Today’s Pipeline authoring is synchronous: writing code, chasing error - every step blocks the engineer until resolved. You can’t step away or parallelize. Airflow Autopilot reimagines this to be AI-native and asynchronous. Describe your pipeline’s intent. The agent takes over - orchestrating two classes of purpose-built tools: tools that generate the DAG code and automate setup, and scorer tools that evaluate it across dimensions: e.g. data discovery, auth, compliance, DAG validation, even end-to-end execution. Every scorer returns a deterministic result and structured, prioritized hints. The agent runs the generate → verify → refine loop — calling scorers, reading hints, fixing code, re-scoring — until every dimension passes. You come back to a PR with DAGs that have been iteratively built, tested, and ready for review. For 10,000+ Airflow users, this shifts the engineer from executor to reviewer: you own the intent and final judgment, the agent owns the execution. Attendees leave with the architecture for an AI-native authoring experience, the principles behind decomposing work into scorer-sized verification units, and what it takes to scale this in production.

Airflow testing today is a patchwork: you can validate code and catch obvious breakage early, but many production failures live in the seams—runtime state, persistence, serialization boundaries, API behavior, and the way a real deployment executes work across components. The fast tools are valuable, yet they don’t fully model Airflow as a system. Meanwhile, the default development posture nudges you toward single-process behavior and away from realistic concurrency and state interactions. The result is a familiar trade: quick feedback vs. meaningful confidence. “Airflow in a Box” is a step toward collapsing that trade—making deeper, more production-relevant tests accessible without requiring a full, heavyweight instance for every iteration. In this talk, we’ll discuss methodology, quantify slickness, and share real code!

Airflow’s callback system has undergone significant architectural changes recently. Originally driven by the introduction of Deadline Alerts, these improvements have far broader implications for how callbacks are defined, where they run, and how reliable they are.

In this talk, I’ll cover the user-facing and provider-facing changes along with a brief look at the significant technical design decisions and internal refactoring behind them, such as a new workload type and unified type-agnostic database model for callbacks. In the long term, this work makes both callbacks and the Dag Processor more robust, and the improved isolation is a key stepping stone toward Airflow’s upcoming multi-team capabilities.

How do you monitor Airflow across 50 teams in real-time? How do downstream systems react instantly to pipeline completions without polling APIs? How do you build custom dashboards without overloading Airflow’s database? This talk demonstrates how we use Change Data Capture to stream Airflow’s metadata to Kafka, making orchestration events consumable by any system in real-time. By capturing changes in Airflow’s Postgres database and publishing them to Kafka topics, we enable instant notifications, real-time dashboards, compliance audit trails, and cross-system orchestration without modifying Airflow code or impacting performance. You’ll learn how to set up Debezium CDC for Airflow’s metadata tables, design Kafka topics for task and DAG events, build real-time consumers for monitoring and alerting, handle schema evolution across Airflow upgrades, and implement cost attribution and SLA monitoring in real-time. Using production examples processing millions of events daily, I’ll share architecture decisions, performance optimizations, and lessons from running CDC at scale. You’ll leave with patterns for making Airflow observable to your entire organization.

Performance issues in Apache Airflow rarely appear as clear failures. Instead, they surface as subtle signals: longer task queue times, slower DAG parsing, scheduler lag, or workers hitting limits as workloads grow.

In this talk, we share lessons from profiling real production deployments across Airflow 2.x and 3.x. Combining frontline operational insights with focused technical investigation, we analysed task latency, DAG parsing time, worker behaviour, and metadata database performance under sustained load.

We show how configuration choices such as parallelism, max active runs, and worker resources can amplify or limit version-level improvements. We also discuss performance drift in long-running environments, where accumulated DAG runs expose slow queries or missing indexes that fresh deployments do not reveal.

Finally, we examine how dynamic DAG generation (e.g. with cosmos dbt dags) and custom user code can unintentionally impact parsing and execution performance.

Attendees leave with a practical framework to profile existing deployments, isolate bottlenecks, optimise performance, reduce recurring issues, and approach upgrades with confidence.

In this session I will provide a deep dive into a task instance’s lifetime. From when the scheduler decides for it to be scheduled until it is marked as success or failed.

We will explore when in the process concepts like concurrency, pools and priority weights apply, what it means for a task to be “queued” and where things like cluster policies, operator links, callbacks and event listeners are evaluated.

The goal is to have a non-technical reference of the inner workings of Airflow applicable to the day-to-day of Dag and Plugin authors.

Airflow’s legacy SLA (Service level agreement) feature let users set a maximum expected duration for a DAG run and receive an email when it was exceeded, but it was inflexible and hard to configure. Deadline Alerts replaced it in 3.1 with a general-purpose system for time-based alerting. Since then, two release cycles have reshaped the feature.

Callbacks now run in supervised subprocesses with access to Connections, Variables, and Assets, which means they can query your infrastructure and respond to problems, not just send a notification. Deadline status is visible in the UI Grid view and DAG run overview. Named deadlines let you attach multiple alerts to a single DAG for different stakeholders. OpenLineage captures deadline events. And fixes for duplicate callbacks under HA schedulers and migration performance have made the feature production-solid.

As one of the developers of Deadline Alerts, I’ll walk through these changes and show callback patterns that take advantage of the new execution model. I’ll close with where the feature is going: deadlines that attach to individual tasks and assets. The end goal is for Deadline Alerts to make time constraints something the scheduler understands and acts on, not just something you get alerted about after the fact.

Asset partitions are a key building block in Expanded Data Awareness. This session explains the core semantics of partition definitions, partition mappings, and backfill behavior in AIP-76. I will show how these pieces fit together in the current design, then discuss where asset partitions can go next, including improvements in authoring ergonomics, observability, and partition-aware workflow capabilities. Attendees will leave with a clear mental model of today’s implementation and a practical view of future direction.

Airflow 3 has officially arrived! If you’re considering an upgrade, this session will equip you with essential migration utilities that facilitate a smooth transition from Airflow 2.x. Attendees will learn the new CLI command, “airflow config lint”, to analyze your configuration files for any removed, deprecated, or renamed elements. This command provides comprehensive feedback and allows for filtering specific sections and options.

During the session, attendees will learn to leverage a set of rigorous Ruff rules - AIR301, AIR302, and AIR303 - crafted to detect migration issues within your codebase automatically. Notably, rule AIR301 flags DAG definitions lacking an explicit schedule argument, a critical update in Airflow 3. Rule AIR302 identifies deprecated functions and removes configuration settings, offering recommended alternatives. Rule AIR303 highlights code that references components now shifted to provider packages, ensuring your integrations are up to date.

Join this session for live demos and practical examples that will empower you to confidently upgrade, minimize downtime, and achieve optimal performance in Airflow 3.

Problem Statement: As our data platform scaled, our shared Airflow 2.9 deployment became a bottleneck with critical challenges: development friction from shared repositories, custom security workarounds, release coordination complexity, data isolation concerns, and cost attribution opacity. When Airflow 3.x launched with hybrid execution support, we restructured our architecture. Following a successful proof of value, we implemented remote execution - enabling teams to run workloads in isolated Kubernetes clusters while maintaining centralized orchestration. This session shares our journey, architectural decisions, and how we leveraged agentic AI to streamline migration and developer experience.

Presentation Details: Join us for a practitioner’s guide to transforming Airflow from a shared bottleneck into scalable execution with multiple Kubernetes clusters. The Journey: Why we moved from Airflow 2.9’s monolithic deployment to Airflow 3.x’s remote execution. The Architecture: Astronomer orchestrating workloads across team-owned Azure Kubernetes clusters. The Innovation: How agentic AI automates DAG development, from code generation to deployment.

Apache Spark’s new Declarative Pipelines (SDP) let engineers define WHAT their data should look like, not HOW to build it. Apache Airflow 3 brings a declarized orchestration model. Together, they eliminate an entire category of boilerplate: the DAG that exists only to babysit a pipeline. This talk walks through building a production Spark SDP pipeline orchestrated by Airflow 3, showing how dependency graphs replace imperative task chains, how testing and recovery patterns change when your pipeline is declarative end-to-end, and what this means for the 80% of data engineering time currently spent on operational plumbing.

In healthcare data, standards are often anything but standard. Every new partner arrives with its own requirements for data exchange spanning FHIR APIs, HL7 feeds, SFTP drops, and custom vendor extracts.

The result? Integration projects that stretch from weeks into months, custom pipelines that only one engineer understands, and implementation teams who are already counting down to your next missed deadline.

This session shows how Airflow can change your approach to managing data transfer for healthcare partners.

We’ll cover how to structure your pipelines with DAGFactory patterns so that you don’t need to treat partner onboarding as a custom engineering effort and where this approach breaks down. We will also cover how you can integrate with tools like OpenMetadata (using Airflow under the hood) to track data assets, so your implementation team knows what is happening without having to ask an engineer.

This session is for data engineers building or maintaining healthcare integration pipelines, and healthcare leaders who don’t want to keep hearing “the next partner integration is a few months away.”

This talk covers migrating a production Airflow platform that orchestrates a large VM fleet — provisioning, OS patching, and decommissioning at high concurrency. This is not a data pipeline — it is infrastructure operations at fleet scale We’ll share workflow patterns that make fleet-scale orchestration possible in Airflow, then cover how we moved from an Airflow 2 monolith — all components on every node with fixed worker counts — to Airflow 3 with independently scalable services, each with its own release cycle. We’ll dig into a silent breaking change in Airflow 3’s XCom behavior: xcom_pull(key=…) without task_ids no longer searches upstream tasks, returning None with no warning. We’ll present three iterations of solving this — from O(n) DAG traversal to a custom XCom backend that restores Airflow 2 semantics with zero DAG code changes — and the design tradeoffs at each stage. Attendees will learn how Airflow powers infrastructure operations beyond data pipelines, how Airflow 3’s XCom silently breaks Airflow 2 workflows, three approaches to the same migration problem, and lessons from running both versions in parallel.

AI agents break the traditional Airflow trust model. While standard tasks are deterministic, agents execute dynamic logic and invoke external tools, meaning untrusted code is suddenly running inside standard containers sharing your host kernel. This session demonstrates how to secure AI workloads in Airflow without rewriting the orchestrator or building custom executors. We will introduce a custom, policy-driven @agent TaskFlow abstraction that leverages Kubernetes executor_config overrides (like runtimeClassName) to isolate workloads on the fly.

Key Takeaways for Attendees:

• The Threat Model: Why containers are not a strong enough security boundary for AI agents.
• The Implementation: How to build an @agent decorator that routes tasks to sandboxed environments (gVisor, Kata, Peer Pods) while keeping the KubernetesExecutor unchanged.
• Kubernetes in Production: How to achieve a VM-per-pod pattern using open-source tools without requiring nested node virtualization.
• Operational Realities: A candid look at execution flow, pod spec mutation, and the latency/cost trade-offs of runtime isolation.

Teams running Airflow on Kubernetes know the trade‑off all too well: Kubernetes scales beautifully in production, but makes local development slow, brittle, and unrealistic. Engineers struggle to replicate production environments locally, forcing them into inefficient “test-in-production” cycles that slow delivery velocity, increase deployment risk, and frustrate data teams.

In this talk, we’ll walk through the architectural patterns and platform engineering approach we used to give engineers on‑demand, isolated, production‑like Airflow environments, without sacrificing the benefits of shared Kubernetes infrastructure.

What You’ll Learn:

• An architectural pattern for provisioning on-demand, isolated Airflow environments on shared EKS infrastructure
• Real-world lessons from operating this solution in production: what worked, what didn’t, and what we’d do differently
• Measurable outcomes: how this approach reduced DAG development cycle time and improved engineer satisfaction

If you’re operating Airflow on Kubernetes—or designing internal platforms for data and ML teams—this session offers a concrete, battle‑tested blueprint to improve Airflow delivery from your stakeholders.

When Airflow 3 introduced JWT based task authentication, it also introduced new attack surfaces; such as, Tokens that can’t be revoked,Tasks that lose authentication while waiting in queues and Forked processes that inherit signing keys and also can forge tokens for other tasks.

In this talk, I’ll walk through three security challenges at the task execution boundary and the code contributed to fix them:

Token revocation (merged, PR #61339): Airflow 3.x had no way to invalidate issued JWTs with implications for common compliance frameworks.

Scope separation (in progress, PR #60108): A two-token mechanism separating long lived workload tokens from short lived execution tokens which is in review with the Airflow core team.

Task identity provenance (in active discussion): I’ll present a proposed defense, a server-side JTI allowlisting that could make forged tokens useless across all execution topologies.

This session is deeply technical and grounded in real contributed code including what attack vectors existed before each fix and the audience will leave understanding Airflow 3’s token security model and practical patterns for securing multi team task execution.

What does solving a Rubik’s Cube have to do with Apache Airflow? More than you’d think.

In this talk, I’ll walk through a project where Airflow orchestrates the process of solving a Rubik’s Cube — not as a gimmick, but as a framework for exploring cyclic workflows, state management, and iterative computation in a system designed for DAGs. Cube-solving algorithms naturally require feedback loops, evolving state, and conditional branching — all things that challenge Airflow’s acyclic model.

We’ll explore how to model “cycles” without breaking DAG semantics, manage cube state across tasks, handle convergence and termination conditions, and avoid common anti-patterns. Along the way, I’ll share practical lessons about idempotency, XCom design, task explosion, and when to rethink orchestration boundaries.

If you’ve ever tried to push Airflow beyond straightforward ETL, this session will give you concrete patterns for safely orchestrating iterative, stateful workflows in production.

As Airflow deployments scale and the number of Dag authors increases the question arises: how do we support many teams with different needs and requirements on a shared platform? Over the years we’ve observed many organizations building their own multi-tenant layers on top of Apache Airflow to solve this problem and we’re now adding native support for this type of deployment. This talk explores building multi-team support in Airflow, working backwards from those real deployment challenges and community pain points we’ve observed.

Rather than strict multi-tenancy with complete isolation, we designed a flexible multi-team architecture allowing isolated task execution, UI experience and connections/variables/secrets management to name a few. This allows multiple teams of engineers to operate within a single Airflow environment, sharing the scheduler API server and database.

We’ll cover some of the technical details of the new architecture, how what we’ve built differs from multi-tenancy, and demo how to use multi-team.

What if your pipeline could tell the difference between recoverable errors and real bugs and handle both without waking anyone up? At OnsiteIQ, we process millions of construction site images monthly through Airflow with mixed AWS Batch spot and on-demand tasks. We need to handle corrupt data, spot evictions, and real bugs. Before Airflow, every failure looked the same: something broke, an engineer investigated, and the same transient infrastructure issues kept masking real bugs underneath. In a 3-month solo migration, I built custom Airflow operators that automatically classified and handled every failure via Airflow’s callbacks. Actual code bugs surface through clean, noise-free alerts directly to actionable tickets. Every genuine bug got caught exactly once and permanently fixed. Engineering oversight dropped from 20% to zero within months. This talk covers the error classification architecture, automatic fallback patterns, and the framework for turning Airflow into a self-healing system.

In many modern data platforms, orchestration tools are combined with transformation frameworks. A common pattern is orchestrating dbt (data build tool) transformations using Apache Airflow — something reported by roughly 44% of the community.

At first glance, the integration seems straightforward: simply run dbt run inside an Airflow task. Some teams go further and use libraries that convert dbt projects into native Airflow DAGs, such as Astronomer Cosmos.

In practice, however, teams quickly run into operational and architectural challenges. Slowness, out-of-memory errors, zombie tasks, and DAGs that take minutes to appear in the UI are just a few of the issues that can emerge as projects scale.

In this talk, I’ll walk through the most common problems I’ve encountered while supporting organisations running dbt with Airflow, explain why they occur, and share practical strategies to avoid or resolve them. The goal is to help teams build more reliable and scalable pipelines when combining Airflow orchestration with dbt transformations.

Airflow’s evolution toward a client-server architecture faced a fundamental challenge: splitting a monolithic codebase into independent distributions (airflow-core, task-sdk, providers) without triggering dependency hell. Traditional PyPi packaging and code duplication both fail at Airflow’s scale.

Airflow 3.2 solves this through modular isolation and shared libraries using in-repository symlinks. This approach ensures each distribution ships with the exact version of shared code it requires, eliminating runtime version conflicts and allowing for independent dependency management. We have already migrated 10+ critical components—including the config parser, observability, and secrets masking—into this shared model.

This architecture unlocks:

Zero Version Conflicts: Mix and match airflow-core and task-sdk versions seamlessly.

Streamlined Maintenance: Automatic security fixes across all components.

AIP-72 Vision: Lightweight workers with API-first communication, removing the need for direct database access.

Join us to explore how shared libraries transform Airflow’s monorepo into a modular, scalable, and truly distributed orchestration platform!

As organizations scale their data platforms, managing access to Apache Airflow becomes increasingly complex. In this talk, we introduce the Keycloak Auth Manager — a pluggable authentication and authorization backend for Airflow that delegates identity management to Keycloak, a battle-tested open-source Identity and Access Management solution.

We’ll start with the big picture: what problem does the Keycloak Auth Manager solve, and why Keycloak? We’ll walk through the architecture — how Airflow’s auth manager interface works, how the Keycloak integration hooks into it, and how authentication flows (OIDC/OAuth2) and authorization (role mapping, resource-based permissions) are handled under the hood.

In the second part, we shift to the user perspective with a demo. You’ll see how to configure and deploy the integration, how end users experience SSO login, and how administrators manage roles and permissions in Keycloak that reflect directly in Airflow’s UI and API.

Finally, We’ll also touch on how Keycloak naturally fits into multi-team scenarios, and what that unlocks for teams operating at scale.

Key takeaways:

• Understand how Airflow’s pluggable auth manager interface works
• Learn how Keycloak handles authentication (OIDC) and authorization (roles/permissions) for Airflow
• See a real-world deployment in action, from config to login to access control
• Walk away with practical tips for adopting this in your own stack

Last year, we showed how LinkedIn’s continuous deployment (LCD) runs on Apache Airflow to orchestrate safe, repeatable releases across thousands of services—powering everyday deployments for 10,000+ engineers.

This year, we’ll dive into the hard‑won patterns that keep those deployments stable at scale: preserving DAG consistency during live updates; routing seamlessly across multiple clusters for graceful failover; enforcing HA guardrails on the control plane; and using dynamic task mapping to deliver faster rollbacks and reduce deployment overhead. You’ll see how we abstract Airflow for a cleaner user experience, what really moved the needle on launching tasks faster, and portable observability practices that cut on‑call toil.

Key takeaways:

• Abstracting Airflow UX: define and update workflows without Airflow internals
• Multi‑cluster routing and failover: keep pipelines running during degradation
• DAG consistency without per‑run versioning: controlled ingestion and safe re‑ingestion
• Dynamic task mapping: faster rollbacks and practical rerun strategies
• Observability: health metrics, alerts, SLOs, and incident playbooks

The rise of more complex asset and agentic powered workflows, the Airflow UI needs to evolve beyond just a way to view failed logs and relationships between tasks.

Come see how we are leveraging the latest Airflow features to build new user experiences that can handle growing agentic workflows.

We’ll go through a few workflows to see how they can be solved through a traditional Dag-centric view or a new Asset-centric view. We will also showcase how both are becoming more realtime so you can always see what is happening.

Come with your own use cases for us to discuss what user experience fits each best.

Airflow running slow? Memory is spiking. Tasks are queuing forever. Now what? Debugging performance issues in a distributed system like Airflow can feel overwhelming—is it the scheduler, the database, the DAG Processor, or your DAG code? This talk shares practical techniques for isolating and fixing performance problems, using real examples from the Airflow codebase.

We’ll cover:

1. Understanding Airflow’s moving parts – Where bottlenecks typically hide (scheduler loop, DAG parsing, database queries).

2. Profiling techniques – Memory profiling, query analysis, and metrics that actually matter.

3. Case study: DAG Processor OOM – How a single SQLAlchemy query caused a memory explosion, and how we fixed it.

4. Testing your fixes – Setting up reproducible performance tests before and after.

Whether you’re troubleshooting your own deployment or contributing fixes upstream, you’ll leave with a toolkit for tackling Airflow performance issues.

This session details Idelic/Descartes’s critical journey to a robust, scaled Astronomer Airflow environment. We’ll share technical lessons from overcoming initial orchestration challenges and successfully scaling to over 1,000 active DAGs. The session will showcase our advanced, Jenkins-integrated testing deployment for managing this scale, and the development of a standardized framework that simplifies DAG creation, eliminates code repetition, and enables configuration changes without a full deployment. This is essential for any team managing complex data pipelines, offering a blueprint for standardized Airflow development, maximum data reliability, and future growth at a large scale.

At Lyft, driver pay configs on GitHub must be validated through Airflow DAGs before merging. However, Scientists and Analysts who change configs are not familiar with Airflow. How do we make such validation self-service while meeting SOX compliance?

This talk presents a design pattern for bidirectional GitHub-Airflow integration: GitHub Actions trigger DAGs, and DAGs push results back as PR status checks via the GitHub Commit Status API. We cover event-driven push-style vs traditional polling style, and why an event-driven push-style works well with Dynamic Task Mapping. This pattern aligns with Airflow 3’s event-driven scheduling vision. We also discuss how SOX requirements shaped this design.

GitHub-Airflow integration is not just hitting APIs. We also address real-world challenges: choosing between GitHub API and GitPython for read/write patterns, preventing DAGs from overriding built-in PR checks, blocking merges during long-running execution, and handling PR rebases that orphan commit statuses. Our solution uses a persistent mapping table to propagate results across rebased commits.

Attendees will leave with a reusable blueprint for event-driven GitHub-Airflow integration.

Orchestrating AI workloads introduces a two-front battle with infrastructure instability. First, the Airflow workers themselves (e.g., Kubernetes pod evictions, Celery node scaling) can restart and lose track of active tasks. Second, the external AI cluster running the heavy compute can experience temporary network blips, API timeouts or compute rescheduling. With standard Dag designs, these transient hiccups often cause Airflow to panic, fail the task, and tragically send a kill signal to an expensive, perfectly healthy AI job.

This Builder Track session explores a specialized Dag design pattern engineered to solve this dual-instability problem entirely at the code level. Rather than managing the underlying infrastructure, we will dive into how to write resilient Airflow tasks that act as fault-tolerant “watchers.” You will learn how to author Dags that survive worker evictions, patiently handle external AI cluster timeouts, and accurately reflect the true state of the workload, ensuring your pipelines remain bulletproof.

With the growing recognition if the need for agentic orchestration, Airflow is evolving to support a growing set of agentic patterns. Dynamic Task mapping provided a foundation for RAG workflows. Learn how to go beyond those and orchestrate reasoning patterns with dynamic graphs

Amazon Prime Video uses Airflow to forecast traffic for hundreds of micro-services to deliver the best customer experience for some of the world’s biggest live events across multiple global regions. The forecasting methodology involves complex job dependencies between customer interaction metrics and geographies - translating to ~50 production DAGs with cross-DAG dependencies that process terabytes of customer activity data daily across tens of thousands of compute cores. In this talk, we’ll cover how we manage dependency complexity at scale, coordinate data flows across geographical boundaries, and keep forecasts reliable as the system grows.

Data Mesh decentralises data ownership across business domains. In regulated industries each domain operates in its own account where producers publish data products and consumers subscribe. This enforces governance, limits blast radius and preserves autonomy. When each domain runs its own Airflow, orchestrating across these boundaries is the central challenge. Airflow 2.4 introduced data-aware scheduling which were designed for single Airflow instance with no native cross-instance event propagation. In practice this meant building polling sensors that queried the producer REST API to check upstream completion, but it is unreliable as events were lost and ordering not guaranteed. Airflow 3.0 resolves this with Event-driven scheduling via AssetWatcher. The Triggerer monitors a message queue and triggers the consumer DAG when the producer publishes a completion event. This talk traces that journey through a regulated enterprise Data Mesh. We also share how we built an agentic AI skills framework that encodes operational Airflow knowledge into reusable skills, enabling an AI agent to autonomously deploy, validate and troubleshoot the cross-environment pattern end-to-end.

This talk focuses on leveraging the Task State Management (AIP-103) and Enhanced Retry Policy work (AIP-105) being released in Airflow 3.3 to enable enhanced execution of long running tasks including checkpointing, sophisticated (and automated) retry policies, and intra-task observability.

Initially focused on Apache Spark, which is one of the most widely used workload frameworks for data engineers, this is extensible to long running tasks of any type including agentic workflows.

This solves a key pain point for Airflow users, without requiring custom async code development.

Building an AI capability in Airflow is the easy part. The hard part is what comes next.

You want to swap a model, refactor a prompt, cut token costs, or try a local model instead of paying for cloud. How do you know it still works as expected? Without a fast feedback loop, every change is a gamble.

This talk shows practical patterns for building that feedback loop, with real examples using agent skills, MCPs, and local and cloud models. It covers the challenges too: sandboxing, observability, non-determinism, and keeping checks simple enough that people actually use them.

In my almost 15 years as a data engineer, I’ve learned one universal truth: everyone needs orchestration. The marketing team needs daily attribution reports. The CRM team needs personalized newsletter triggers. The platform team needs cross-cloud data transfers. The analytics team needs third-party data imports. Data touches every corner of the business, and the orchestration layer is the one layer that connects it all.

This talk explores what becomes possible when we decouple pipeline logic (what happens) from definition (how it’s authored). With the right abstractions, the authoring interface can be anything: Python, declarative YAML, templates, spreadsheets, or even a video game.

We will explore different levels of orchestration abstraction: native Python, declarative config (using the open-source project DAG Factory), templates (using the open-source project Blueprint), and AI-assisted Authoring. To prove the power of this decoupled architecture, I will showcase a custom Minecraft plugin that lets you build Airflow Dags by placing blocks in Minecraft (yes, you heard that right).

Apache Airflow’s Kubernetes integration enables flexible workload execution on Kubernetes but lacks advanced resource management features including application queueing, tenant isolation and gang scheduling. These features are increasingly critical for data engineering as well as AI/ML use cases, particularly GPU utilization optimization. For example, gang scheduling ensures all required resources for a job are allocated atomically, preventing partial allocations that waste resources. Apache Yunikorn, a Kubernetes-native scheduler, addresses these gaps by offering a high-performance alternative to Kubernetes default scheduler. In this talk, we’ll demonstrate how to conveniently leverage Yunikorn’s power in Airflow, along with practical use cases and examples.

Meet airflowctl, the new default for API-driven remote operations. You will see how separating control from execution enhances security, enables isolation, and simplifies automation across different environments. I will discuss the development of airflowctl, demonstrate practical examples of secure remote execution, and provide a guide for transitioning from legacy workflows. You will learn how to easily migrate towards airflowctl and leverage the flexibility of an API-driven approach.

Your data platform team didn’t sign up to be a Dag factory. But when Airflow expertise is concentrated in a small group of engineers, that’s exactly what happens. Analysts wait days for simple workflows, engineers burn cycles rebuilding the same patterns, and frustrated teams start building outside the stack entirely. The real fix isn’t a better onboarding guide or a friendlier UI. It’s rethinking the abstraction layer your team exposes to the rest of the business.

In this talk, we’ll introduce astronomer/blueprint, an open-source Python library built around the idea that platform teams should define the building blocks and everyone else should be able to assemble them. We’ll walk through how to design composable, type-safe templates using Pydantic validation and control exactly which parameters downstream users can configure. We’ll also show how we’ve layered a no-code visual interface on top in Astro, Astronomer’s unified orchestration platform for Apache Airflow®, giving non-technical users a path to self-service without sacrificing governance or control. You’ll leave with a concrete framework and open-source tooling you can put to work immediately.

A live text adventure where Airflow is the game engine. Rooms are tasks. Choices are branches. Inventory lives in XComs. Monsters have Deadline Alerts. The audience votes at every fork, and the DAG decides what happens next. It’s silly, it’s live, and every concept maps to a real production pattern.

No two open source projects have shaped modern data engineering more than Apache Spark and Apache Airflow. But their partnership wasn’t designed- it was earned. From the early days of BashOperator wrapping spark-submit, through the SparkSubmitOperator, Livy, Kubernetes-native execution, and now Airflow 3’s asset-aware scheduling paired with Spark’s Declarative Pipelines, the integration story is a masterclass in how independent communities converge on shared problems without shared governance. This talk traces the full arc: how Spark’s compute model and Airflow’s orchestration model co-evolved, where they fought, where they complemented each other, and what the next chapter looks like as both projects ship their most ambitious releases simultaneously. Along the way, we’ll examine the contribution patterns, the cross-pollination of committers, and why this particular pairing outlasted every managed alternative that tried to replace it. This is not a vendor talk. This is a community talk about what happens when two ecosystems trust each other enough to stay independent.

Building on Airflow 3’s new worker structure and foundation laied by Go SDK, we take a look at how Airflow can support a fully cross-language Dag-authoring experience.

We will discuss how a new language SDK is built, how a task talks to Airflow, and how multiple languages may be mixed inside a Dag. To support additional languages without logic duplication, a new middle layer is required between Airflow and the task. Additional topics, such as security, distributed workload, and user interface considerations, will also be touched on.

Datadog is a world-class data platform ingesting more than a 100 trillion events a day, providing real-time insights. Since our internal adoption of Airflow following the release of 3.0.0, the number of teams relying on our internal Airflow platform have grown organically and quickly.

This internal Airflow adoption came with a number of platform challenges, requiring novel solutions which could support multi-tenancy, scalability, and bespoke runtime environments. In this talk, we will cover how we’ve expanded the functionality of Airflow triggers – via trigger queue assignment – to support multi-tenancy deployments, while contributing those solutions upstream with the broader Airflow community. We’ll cover the conceptual design and motivations for Trigger queues, and how the trigger queue pattern can benefit both multi-tenant and single-occupant Airflow systems alike.

Airflow is Python-first — but production business logic is often Java-first. The Airflow Java SDK bridges that gap by letting you mix Java and Python tasks within the same DAG, without shell wrappers or separate services.

In this talk, we’ll walk through the full lifecycle of a Java task: how the Java SDK is set up, how tasks are defined and packaged as a JAR, how Airflow picks them up and runs them on any executor, and how results flow back into your DAG. We’ll also cover how core Airflow primitives — Variables, Connections, XCom, and logging — work natively in the Java SDK, enabling true cross-language, bidirectional communication within a single pipeline. You’ll see it all running end-to-end in a live demo alongside Python tasks.

If you work with Java services and have ever wished you could orchestrate them without translating everything into Python first, this talk is for you.

Migrating from UC4 Automic to Apache Airflow is far from a lift-and-shift exercise. UC4 offers advanced scheduling primitives that data teams rely on daily — and Airflow doesn’t replicate them out of the box.

At eBay, we migrated thousands of business-critical UC4 workflows onto our Airflow 2.10 platform. Rather than forcing teams to change how they operate, we built the missing capabilities natively into Airflow:

• Breakpoints — pause a pipeline at a specific task for inspection without failing the run
• Skip logic — dynamically bypass tasks or task groups at runtime
• Calendar-aware scheduling — replicate UC4’s calendar model as custom Airflow timetables
• Pipeline pause/resume — operator-triggered suspension of in-flight DAG runs with state consistency

We’ll share the engineering trade-offs, architectural constraints we hit, and patterns reusable beyond eBay’s stack.

As Airflow adoption expands across large enterprises, a core challenge emerges: How to enable multiple teams to design and operate data pipelines without relying heavily on specialized engineering expertise. In this session, we will present a zero‑code, metadata‑driven Airflow framework built and deployed within a large financial services organization to accelerate pipeline development and onboarding at scale.

This framework allows users to define workflows using simple CSV or Excel inputs, which are automatically converted into YAML configurations and deployed as fully production‑ready Airflow DAGs using standardized templates on Astronomer. By leveraging a remote execution model and reusable DAG patterns, the solution supports orchestration across heterogeneous systems—including data warehouses, ingestion pipelines, and data quality frameworks—while maintaining enterprise‑grade governance, consistency, and observability.

The talk will walk through the high-level architecture, including Excel‑to‑YAML transformation logic, dynamic DAG generation patterns, and controls that enable non‑developer and cross‑functional teams to safely create and manage pipelines with minimal coding. We will also share lessons learned from taking this framework from initial design to enterprise‑wide production rollout, highlighting how it reduced onboarding time, enforced standardization, and scaled orchestration across teams.

Attendees will gain practical insights into implementing low‑code and no‑code orchestration on top of Apache Airflow, along with key architectural considerations for operating Airflow at enterprise scale.