# Airflow Summit 2026 | Aug. 31 - Sep 2 in Austin, TX

Base URL: https://airflowsummit.org/

> Airflow Summit is the premier conference for the worldwide community of developers and users of Apache Airflow.

# SECTION: Sessions

This is the archive of sessions given in the different editions of Airflow Summit.



## Keynote: Airflow then and now
[URL](https://airflowsummit.org/sessions/airflow-then-now/)  
Date: 2020-07-06T16:00:00.000Z
Summary:
Bolke and Maxime tell us about past on current time of Apache Airflow.


## Airflow at Société Générale : An open source orchestration solution in a banking environment
[URL](https://airflowsummit.org/sessions/airflow-societe-generale/)  
Date: 2020-07-06T17:00:00.000Z
Summary:
This talk covers an overview of Airflow as well as lessons learned of its implementation in a banking production environment which is Société Générale. It will be the summary of a two-year experience, a storytelling of an adventure within Société Générale in order to offer an internal cloud solution based on Airflow (AirflowaaS).


## Scheduler as a service - Apache Airflow at EA Digital Platform
[URL](https://airflowsummit.org/sessions/scheduler-as-a-service/)  
Date: 2020-07-06T18:00:00.000Z
Summary:
In this talk, we share the lessons learned while building a scheduler-as-a-service leveraging Apache Airflow to achieve improved stability and security for one of the largest gaming companies. The platform integrates with different data sources and meets varied SLA’s across workflows owned by multiple game studios. In particular, we present a comprehensive self-serve airflow architecture with multi-tenancy, auto-dag generation, SSO-integration with improved ease of deployment.


## Keynote: How large companies use Airflow for ML and ETL pipelines
[URL](https://airflowsummit.org/sessions/how-airbnb-twitter-lyft-use-airflow/)  
Date: 2020-07-07T16:00:00.000Z
Summary:
In this talk, colleagues from Airbnb, Twitter and Lyft share details about how they are using Apache Airflow to power their data pipelines.


## Data DAGs with lineage for fun and for profit
[URL](https://airflowsummit.org/sessions/data-dags-with-lineage/)  
Date: 2020-07-07T17:00:00.000Z
Summary:
Let’s be honest about it. Many of us don’t consider data lineage to be cool. But what if lineage would allow you to write less boilerplate and less code, while at the same time make your data scientists, your auditors, your management and well everyone more happy? What if you could write DAGs that mix between tasks based and data based?


## Airflow on Kubernetes: Containerizing your workflows
[URL](https://airflowsummit.org/sessions/airflow-kubernetes/)  
Date: 2020-07-07T18:00:00.000Z
Summary:
At Nielsen Digital we have been moving our ETLs to containerized environments managed by Kubernetes. We have successfully transferred some of our ETLs to this environment in production. In order to do this we used the following technologies: Helm to easily deploy Airflow on to Kubernetes; Airflow’s Kubernetes Executor to take full advantage Kubernetes features; and Airflow’s Kubernetes Pod Operator in order to execute our containerized Tasks within our DAGs. To automate a lot of the deployment process we also used Terraform. Lastly, Kubernetes features were used to gain much more fine grained control of Airflows infrastructure.


## Data flow with Airflow @ PayPal
[URL](https://airflowsummit.org/sessions/data-flow-with-airflow-at-paypal/)  
Date: 2020-07-08T04:00:00.000Z
Summary:
In PayPal we decided to move away from two of our enterprise schedulers, Control-M and UC4, to Airflow. As we started the journey, the first most important step that we wanted to take was to build all the mandatory API’s on the top of Airflow so that we could integrate with our Self-Service Tools. In this talk we share the challenges that we ran into while building APIs on top of Airflow and how we overcame them.


## Democratised data workflows at scale
[URL](https://airflowsummit.org/sessions/democratised-data-workflows-at-scale/)  
Date: 2020-07-08T05:00:00.000Z
Summary:
Financial Times is increasing its digital revenue by allowing business people to make data-driven decisions. Providing an Airflow based platform where data engineers, data scientists, BI experts and others can run language agnostic jobs was a huge swing. One of the most successful steps in the platform’s development was building our own execution environment, allowing stakeholders to self deploy jobs without cross team dependencies on top of the unlimited scale of Kubernetes. In this talk we share how we have integrated and extended Airflow at Financial Times.


## Migrating Airflow-based Spark jobs to Kubernetes - the native way
[URL](https://airflowsummit.org/sessions/migrating-airflow-based-spark-jobs-to-kubernetes/)  
Date: 2020-07-08T06:00:00.000Z
Summary:
At Nielsen Identity Engine, we use Spark to process 10’s of TBs of data. Our ETLs, orchestrated by Airflow, spin-up AWS EMR clusters with thousands of nodes per day. In this talk, we’ll guide you through migrating Spark workloads to Kubernetes with minimal changes to Airflow DAGs, using the open-sourced GCP Spark-on-K8s operator and the native integration we recently contributed to the Airflow project.


## Keynote: Future of Airflow
[URL](https://airflowsummit.org/sessions/future-airflow/)  
Date: 2020-07-08T16:00:00.000Z
Summary:
A team of core committers explain what is coming to Airflow 2.0.


## Run Airflow DAGs in a secure way
[URL](https://airflowsummit.org/sessions/run-airflow-dags-secure-way/)  
Date: 2020-07-08T17:00:00.000Z
Summary:
In the contemporary world security is important more than ever - Airflow installations are no exception. Google Cloud Platform and Cloud Composer offer useful security options for running your DAGs and tasks in a way so you effectively can manage a risk of data exfiltration and access to the system is limited.


## Airflow as the next gen of workflow system at Pinterest
[URL](https://airflowsummit.org/sessions/airflow-as-next-gen-workflow-at-pinterest/)  
Date: 2020-07-08T18:00:00.000Z
Summary:
At Pinterest, our current workflow system, called pinball, has served the data pipeline orchestration demands well for years. However, with the rapid increasing execution demand the system started to expose scalability and performance issues. Therefore we decided to look for a new solution to better address the issues and serve the workflow scheduling demand, and we chose Airflow as our next generation of workflow. In this talk we discuss how we made the decision to on board to Apache Airflow, and beyond the out-of-box features and experience what improvements we made to better support the business need at Pinterest.


## Keynote: Making Airflow a sustainable project through D&I
[URL](https://airflowsummit.org/sessions/making-airflow-community-sustainable-through-di/)  
Date: 2020-07-09T16:00:00.000Z
Summary:
Gris Cuevas shares some statistics about the state of D&I at the Apache Software Foundation and also the initiative the foundation is taking to make projects more diverse and inclusive. Then, Aizhamal shares her own journey on becoming an open source contributor, and dives into project specific initiatives that help Apache Airflow to be one of the most sustainable projects in open source.


## Improving Airflow's user experience
[URL](https://airflowsummit.org/sessions/improving-airflow-user-experience/)  
Date: 2020-07-09T17:00:00.000Z
Summary:
Astronomer is focused on improving Airflow’s user experience through the entire lifecycle — from authoring + testing DAGs, to building containers and deploying the DAGs, to running and monitoring both the DAGs and the infrastructure that they are operating within — with an eye towards increased security and governance as well. In this talk we walk you through some current UX challenges, an overview of how the Astronomer platform addresses the major challenges, and also provide sneak peek of the things that we’re working on in the coming months to improve Airflow’s user experience.


## Airflow CI/CD: Github to Cloud Composer (safely)
[URL](https://airflowsummit.org/sessions/airflow-cicd/)  
Date: 2020-07-09T18:00:00.000Z
Summary:
Deploying bad DAGs to your Airflow environment can wreak havoc. This talk provides an opinionated take on a mono repo structure for GCP data pipelines leveraging BigQuery, Dataflow and a series of CI tests for validating your Airflow DAGs before deploying them to Cloud Composer.


## Advanced Apache Superset for Data Engineers
[URL](https://airflowsummit.org/sessions/advanced-apache-superset/)  
Date: 2020-07-10T04:00:00.000Z
Summary:
Superset is the leading open source data exploration and visualization platform. In this talk, we’ll be presenting Superset with a focus on advanced topics that are most relevant to Data Engineers. The presentation will be largely a live demo of the product, with a deeper dive into advanced topics for Data Engineers.


## Teaching an old DAG new tricks
[URL](https://airflowsummit.org/sessions/teaching-old-dag-new-tricks/)  
Date: 2020-07-10T05:00:00.000Z
Summary:
Scribd is migrating its data pipeline from an in house system to Airflow. It’s a one big giant data pipeline consisting of more than 1,500 tasks. In this talk, I would like to share couple best practices on setting up a cloud native Airflow deployment in AWS. For those who are interested in migrating a non-trivial data pipeline to Airflow, I will also share how Scribd plans and executes the migration.


## Ask me anything with Airflow members
[URL](https://airflowsummit.org/sessions/ama/)  
Date: 2020-07-10T06:00:00.000Z
Summary:
We will host an ‘ask me anything’ with a group of Airflow committers & PMC members.

## Demo: Reducing the lines, a visual DAG editor
[URL](https://airflowsummit.org/sessions/demo-visual-dag-editor/)  
Date: 2020-07-10T16:00:00.000Z
Summary:
In this talk I will introduce a DAG authoring and editing tool for Airflow that we have built. Installed as a plugin, this tool allows users to author DAGs compose existing operators and hooks with virtually no Python experience. We walk through a demo of DAG authorship and deployment, and spend time reviewing the underlying open-source standards used and the general approach that was taken to develop the code.


## AIP-31: Airflow functional DAG definition
[URL](https://airflowsummit.org/sessions/aip-31-airflow-functional-dag-definition/)  
Date: 2020-07-10T17:00:00.000Z
Summary:
Airflow does not currently have an explicit way to declare messages passed between tasks in a DAG. XCom are available but are hidden in execution functions inside the operator. AIP-31 proposes a way to make this message passing explicit in the DAG file and make it easier to reason about your DAG behaviour.


## Using Airflow to speed up development of data intensive tools
[URL](https://airflowsummit.org/sessions/using-airflow-speed-development-data-tools/)  
Date: 2020-07-10T18:00:00.000Z
Summary:
In this talk we review how Airflow helped create a tool to detect data anomalies. Leveraging Airflow for process management, database interoperability, and authentication created an easy path forward to achieve scale, decrease the development time and pass security audits. While Airflow is generally looked at as a solution to manage data pipelines, integrating tools with Airflow can also speed up development of those tools.


## Autonomous driving with Airflow
[URL](https://airflowsummit.org/sessions/autonomous-driving-airflow/)  
Date: 2020-07-13T16:00:00.000Z
Summary:
This talk describes how Airflow is utilized in an Autonomous driving project, originating from Munich - Germany. We describe the Airflow setup, what challenges we encountered and how we maneuvered to achieve a distributed and highly scalable Airflow setup.


## From cron to Airflow on Kubernetes: A startup story
[URL](https://airflowsummit.org/sessions/cron-airflow-kubernetes-startup-story/)  
Date: 2020-07-13T17:00:00.000Z
Summary:
Learn how Devoted Health went from cron jobs to Airflow deployment Kubernetes using a combination of open source and internal tooling.


## Airflow in Airbnb
[URL](https://airflowsummit.org/sessions/airflow-airbnb/)  
Date: 2020-07-13T18:00:00.000Z
Summary:
Go over the yesterday, today and tomorrow for Airflow in Airbnb. Share our learnings and vision in Airflow core and around Airflow in its eco system.


## Pipelines on pipelines: Agile CI/CD workflows for Airflow DAGs
[URL](https://airflowsummit.org/sessions/pipelines-agile-ci-cd-workflows/)  
Date: 2020-07-14T04:00:00.000Z
Summary:
How do you create fast and painless delivery of new DAGs into production? When running Airflow at scale, it becomes a big challenge to manage the full lifecycle around your pipelines; making sure that DAGs are easy to develop, test, and ship into prod. In this talk, we will cover our suggested approach to building a proper CI/CD cycle that ensures the quality and fast delivery of production pipelines.


## Production Docker image for Apache Airflow
[URL](https://airflowsummit.org/sessions/production-docker-image/)  
Date: 2020-07-14T05:00:00.000Z
Summary:
This talk will guide you trough internals of the official Production Docker Image of Airflow. It will show you the foreseen use cases for it and how to use it in conjunction with the Official Helm Chart to make your own deployments.


## Airflow as an elastic ETL tool
[URL](https://airflowsummit.org/sessions/airflow-elastic-etl-tool/)  
Date: 2020-07-14T06:00:00.000Z
Summary:
In search of a better, modern, simplistic method of managing ETL’s processes and merging them with various AI and ML tasks, we landed on Airflow. We envisioned a new user friendly interface that can leverage dynamic DAG’s and reusable components to build an ETL tool that requires virtually no training.


## How do we reason about the reliability of our data pipeline in Wrike
[URL](https://airflowsummit.org/sessions/reliability-data-pipeline-wrike/)  
Date: 2020-07-14T16:00:00.000Z
Summary:
In this talk we will share some of the lessons we have learned after using Airflow for a couple of years and growing from 2 users to 8 teams. We cover: establishing a reliable review process on AirFlow, managing multiple Airflow configurations, data versioning.


## Achieving Airflow observability with Databand
[URL](https://airflowsummit.org/sessions/achieving-observability-with-databand/)  
Date: 2020-07-14T17:00:00.000Z
Summary:
While Airflow is a central product for data engineering teams, it’s usually one piece of a bigger puzzle. The vast majority of teams use Airflow in combination with other tools like Spark, Snowflake, and BigQuery. Making sure pipelines are reliable, detecting issues that lead to SLA misses, and identifying data quality problems requires deep visibility into DAGs and data flows. Join this session to learn how Databand’s observability system makes it easy to monitor your end-to-end pipeline health and quickly remediate issues.


## From S3 to BigQuery - How a first-time Airflow user successfully implemented a data pipeline
[URL](https://airflowsummit.org/sessions/s3-bigquery-implementation-data-pipeline/)  
Date: 2020-07-14T18:00:00.000Z
Summary:
BigQuery is GCP’s serverless, highly scalable and cost-effective cloud data warehouse that can analyze petabytes of data at super fast speeds. Amazon S3 is one of the oldest and most popular cloud storage offerings. Folks with data in S3 often want to use BigQuery to gain insights into their data. Using Apache Airflow, they can build pipelines to seamlessly orchestrate that connection. In this talk, Leah walks through how they created an easily configurable pipeline to extract data.


## Building reuseable and trustworthy ELT pipelines (A templated approach)
[URL](https://airflowsummit.org/sessions/building-reusable-trustworthy/)  
Date: 2020-07-15T16:00:00.000Z
Summary:
To improve automation of data pipelines, I propose a universal approach to ELT pipeline that optimizes for data integrity, extensibility, and speed to delivery. The workflow is built using open source tools and standards like Apache Airflow, Singer, Great Expectations, and DBT.


## Testing Airflow workflows - ensuring your DAGs work before going into production
[URL](https://airflowsummit.org/sessions/testing-airflow-workflows/)  
Date: 2020-07-15T17:00:00.000Z
Summary:
How do you ensure your workflows work before deploying to production? In this talk I’ll go over various ways to assure your code works as intended - both on a task and a DAG level.


## Adding an executor to Airflow: A contributor overflow exception
[URL](https://airflowsummit.org/sessions/adding-executor-airflow/)  
Date: 2020-07-15T18:00:00.000Z
Summary:
Engaging with a new community is a common experience in OSS development. There are usually expectations held by the project about the contributor’s exposure to the community, and by the contributor about interactions with the community. When these expectations are misaligned, the process is strained. In this talk Vanessa discusses a real life experience that required communication, persistence, and patience to ultimately lead to a positive outcome.


## Migration to Airflow backport providers
[URL](https://airflowsummit.org/sessions/migration-to-airflow-backport-providers/)  
Date: 2020-07-16T04:00:00.000Z
Summary:
In this talk Anita showcases how to use the newly released Airflow Backport Providers.


## From Zero to Airflow: bootstrapping a ML platform
[URL](https://airflowsummit.org/sessions/bootstrapping-ml-platform/)  
Date: 2020-07-16T05:00:00.000Z
Summary:
At Bluevine we use Airflow to drive our ML platform. In this talk, Noam presents the challenges and gains we had at transitioning from a single server running Python scripts with cron to a full blown Airflow setup. This includes: supporting multiple Python versions, event driven DAGs, performance issues and more!


## Airflow the perfect match in our analytics pipeline
[URL](https://airflowsummit.org/sessions/k3/)  
Date: 2020-07-16T06:00:00.000Z
Summary:
For three years we at LOVOO, a market-leading dating app, have been using the Google Cloud managed version of Airflow, a product we’ve been familiar with since its Alpha release. We took a calculated risk and integrated the Alpha into our product, and, luckily, it was a match. Since then, we have been leveraging this software to build out not only our data pipeline, but also boost the way we do analytics and BI.


## Data engineering hierarchy of needs
[URL](https://airflowsummit.org/sessions/data-engineering-hierarchy-needs/)  
Date: 2020-07-16T16:00:00.000Z
Summary:
Data Infrastructures look differently between small, mid, and large sized companies. Yet, most content out there is for large and sophisticated systems. And almost none of it is on migrating a legacy, on-prem, databases over to the cloud. In order to better explain the evolving needs of data engineering organizations, we will review the hierarchy of needs for data engineering.


## What open source taught us about business
[URL](https://airflowsummit.org/sessions/what-open-source-taught-us-about-business/)  
Date: 2020-07-16T17:00:00.000Z
Summary:
This talk shares Polidea’s journey from mobile app development studio to an OSS oriented business partner. We will tell you our story towards code leadership throughout the years. We are also going to share the challenges and practical insights into managing open source projects in our company. After this talk, you will know how we approached combining open source, business and team management not forgetting about a human aspect.


## Effective Cross-DAG dependency
[URL](https://airflowsummit.org/sessions/effective-cross-dag-dependency/)  
Date: 2020-07-16T18:00:00.000Z
Summary:
Cross-DAG dependency may reduce cohesion in data pipelines and, without having an explicit solution in Airflow or in a third-party plugin, those pipelines tend to become complex to handle. That is the reason we, at QuintoAndar, have created an intermediate DAG to handle relationships across data pipelines called Mediator, in order for them to be scalable and maintainable by any team.


## Airflow: A beast character in the gaming world
[URL](https://airflowsummit.org/sessions/airflow-beast-character-gaming-world/)  
Date: 2020-07-17T16:00:00.000Z
Summary:
Being a pioneer for the past 25 years, SONY PlayStation has played a vital role in the Interactive Gaming Industry. Over 100+ million monthly active users, 100+ million PS-4 console sales along with thousands of game development partners across the globe, big-data problem is quite inevitable. This presentation talks about how we scaled Airflow horizontally which has helped us building a stable, scalable and optimal data processing infrastructure powered by Apache Spark, AWS ECS, EC2 and Docker.


## Machine Learning with Apache Airflow
[URL](https://airflowsummit.org/sessions/machine-learning-airflow/)  
Date: 2020-07-17T17:00:00.000Z
Summary:
This talk discusses how to build an Airflow based data platform that can take advantage of popular ML tools (Jupyter, Tensorflow, Spark) while creating an easy-to-manage/monitor


## Achieving Airflow Observability
[URL](https://airflowsummit.org/sessions/achieving-observability/)  
Date: 2020-07-17T18:00:00.000Z
Summary:
Identify issues in a fraction of the time and streamline root cause analysis for your DAGs. Airflow is the leading orchestration platform for data engineers. But when running Airflow at production scale, many teams have bigger needs for monitoring jobs, creating the right level of alerting, tracking problems in data, and finding the root cause of errors. In this talk we will cover our suggested approach to gaining Airflow observability so that you have the visibility you need to be productive.


## Contributing to Apache Airflow | Journey to becoming Airflow's leading contributor
[URL](https://airflowsummit.org/sessions/2021/contributing-journey-becoming-leading-contributor/)  
Date: 2021-07-08T16:00:00.000Z
Summary:
From not knowing Python (let alone Airflow), and from submitting the first PR that fixes typo to becoming Airflow Committer, PMC Member, Release Manager, and #1 Committer this year, this talk walks through Kaxil’s journey in the Airflow World.
The second part of this talk explains:
how you can also start your OSS journey by contributing to Airflow Expanding familiarity with a different part of the Airflow codebase Continue committing regularly & steadily to become Airflow Committer. (including talking about current Guidelines of becoming a Committer) Different mediums of communication (Dev list, users list, Slack channel, Github Discussions etc) 

## Contributing to Apache Airflow: First Steps
[URL](https://airflowsummit.org/sessions/2021/contributing-first-steps/)  
Date: 2021-07-08T17:00:00.000Z
Summary:
Learn to contribute to the Apache Airflow ecosystem both with and without code. Post an article to the Airflow blog, improve documentation, or dive head-first into into Airflow’s free and open source software community.


## Discussion panel: Keep your Airflow secure
[URL](https://airflowsummit.org/sessions/2021/panel-airflow-security/)  
Date: 2021-07-08T17:30:00.000Z
Summary:
You might have heard some recent news about ransomware attacks for many companies. Quite recently the U. S. Department of Justice has elevated the priority of investigations of ransomware attacks to the same level as terrorism. Certainly security aspects of running software and so called “supply-chain attacks” have made a press recently.
Also, you might have read recently about security researcher who made USD 13,000 via bounties by finding and contacting companies that had old, un-patched versions of Airflow - even if the ASF security process was great and PMC of Airflow has fixed those long time ago.


## You don’t have to wait for someone to fix it for you
[URL](https://airflowsummit.org/sessions/2021/dont-have-to-wait/)  
Date: 2021-07-08T18:20:00.000Z
Summary:
Rachael, a new Airflow contributor, and Leah, an experienced Airflow contributor, share the story of Rachael’s first contribution, highlighting the importance of contributions from new users and the positive impact that non-code contributions have in an open source community.


## Airflow as the Foundation of a Multi-Faceted Data Platform
[URL](https://airflowsummit.org/sessions/2021/airflow-as-the-foundation-multi-faceted-data-platform/)  
Date: 2021-07-09T16:00:00.000Z
Summary:
A discussion with Jay Sen, Data Platform Architect at Paypal, and Ry Walker, Founder/CTO of Astronomer about the central role Airflow plays within Paypal’s data platform, and the opportunity to build stronger integrations between Airflow and other tools that surround it.


## Workshop: Contributing to Apache Airflow
[URL](https://airflowsummit.org/sessions/2021/workshop-contributing-apache-airflow/)  
Date: 2021-07-09T16:00:00.000Z
Summary:
Learn how to become a code contributor to the Apache Airflow project.

## Apache Airflow at Apple - Multi-tenant Airflow and Custom Operators
[URL](https://airflowsummit.org/sessions/2021/apache-airflow-at-apple/)  
Date: 2021-07-09T16:50:00.000Z
Summary:
Running a platform where different business units at Apple can run their workloads in isolation and share operators.


## Lessons Learned while Migrating Data Pipelines from Enterprise Schedulers to Airflow
[URL](https://airflowsummit.org/sessions/2021/lessons-learned-while-migrating-data-pipelines/)  
Date: 2021-07-09T17:00:00.000Z
Summary:
Digital transformation, application modernization, and data platform migration to the cloud are key initiatives in most enterprises today. These initiatives are stressing the scheduling and automation tools in these enterprises to the point that many users are looking for better solutions. A survey revealed that 88% of users believe that their business will benefit from an improved automation strategy across technology and business. Airflow has an excellent opportunity to capture mindshare and emerge as the leading solution here. At Unravel, we are seeing the trend where many of our enterprise customers are at various stages of migrating to Airflow from their enterprise schedulers or ETL/ELT orchestration tools like Autosys, Informatica, Oozie, Pentaho, and Tidal.


## Airflow Journey @SG
[URL](https://airflowsummit.org/sessions/2021/airflow-journey-sg/)  
Date: 2021-07-09T17:30:00.000Z
Summary:
This talk will cover the adoption journey (Technical Challenges & Team Organization) of Apache Airflow (1.8 to 2.0) at Societe Generale.
Time line of events:
POC with v1.8 to convince our management. Shared infrastructure with v1.10.2. Multiple Infrastructure with v1.10.12. On demand service offer with v2.0 (Challenges & REX) 

## Building an Elastic Platform Using Airflow Uniquely as an Orchestrator
[URL](https://airflowsummit.org/sessions/2021/building-an-elastic-platform-using-airflow-orchestrator/)  
Date: 2021-07-09T18:00:00.000Z
Summary:
At QuintoAndar we seek automation and scalability in our data pipelines and believe that Airflow is the right tool for giving us exactly what we need. However, having all concerns mapped and tooling defined doesn’t necessarily mean success.
For months we had struggled with a misconception that Airflow should act as an orchestrator and executor within a monolithic strategy. That could not be further from the truth because of the rise of scalability and performance issues, infrastructure and maintainability costs, and multi-directional impact throughout development teams.


## Data Pipeline HealthCheck for Correctness, Performance, and Cost Efficiency
[URL](https://airflowsummit.org/sessions/2021/pipeline-healthcheck/)  
Date: 2021-07-09T18:30:00.000Z
Summary:
We are witnessing a rapid growth in the number of mission-critical data pipelines that leaders of data products are responsible for. “Are your data pipelines healthy?” This question was posed to more than 200 leaders of data products from various industries. The answers ranged from “unfortunately, no” to “they are mostly fine, but I am always afraid that something or the other will cause a pipeline to break”.
This talk presents the concept of Pipeline HealthCheck (PHC) which enables leaders of data products to have high confidence in the correctness, performance, and cost efficiency of their data pipelines. More importantly, PHC enables leaders of data products as well as their development and operations teams to have high confidence in their ability to quickly detect, troubleshoot, and fix problems that make data pipelines unhealthy. The talk also includes a demo of how PHC helps handle common problems in data pipelines like incorrect results, missing SLAs, and overshooting cost budgets.


## Looking ahead: What comes after Airflow 2.0?
[URL](https://airflowsummit.org/sessions/2021/looking-ahead-what-comes-after-airflow-2/)  
Date: 2021-07-12T16:00:00.000Z
Summary:
(No summary available)

## Airflow: The Power of Stitching Services Together
[URL](https://airflowsummit.org/sessions/2021/airflow-google/)  
Date: 2021-07-12T17:00:00.000Z
Summary:
Apache Airflow is known to be a great orchestration tool that enables use cases that would not be possible otherwise. One of the great features that Airflow has is the possibility to “glue” together totally separate services to establish bigger functionalities. In this talk you will learn about various Airflow usages that let Airflow users to automate their critical company processes and even establish businesses. The examples provided will be based on Airflow used in the context of Cloud Composer which is a managed service to provision and manage Airflow instances.


## Pinterest’s Migration Journey
[URL](https://airflowsummit.org/sessions/2021/pinterest-migration-journey/)  
Date: 2021-07-12T18:00:00.000Z
Summary:
Last year, we were able to share why we have selected Airflow to be our next generation workflow system. This year, we will dive into the journey of migrating over 3000+ workflows and 45000+ tasks to Airflow. We will discuss the infrastructure additions to support such loads, the partitioning and prioritization of different workflow tiers defined in house, the migration tooling we built to get users to onboard, the translation layers between our old DSLs and the new, our internal k8s executor to leverage Pinterest’s kubernetes fleet, and more. We want to share the challenges both technically and usability wise to get such large migrations over the course of a year, and how we overcame it to successfully migrate 100% of the workflows to our inhouse workflow platform branded Spinner.


## The Newcomer's Guide to Airflow's Architecture
[URL](https://airflowsummit.org/sessions/2021/guide-airflow-architecture/)  
Date: 2021-07-12T19:00:00.000Z
Summary:
Airflow has a lot of moving parts, and it can be a little overwhelming as a new user - as I was not too long ago.
Join me as we go though Airflow’s architecture at a high level, explore how DAGs work and run, and look at some of the good, the bad, and the unexpected things lurking inside.


## Dataclasses as Pipeline Definitions in Airflow
[URL](https://airflowsummit.org/sessions/2021/dataclasses-as-pipeline-definitions/)  
Date: 2021-07-13T04:00:00.000Z
Summary:
We will describe how we were able to build a system in Airflow for MySQL to Redshift ETL pipelines defined in pure Python using dataclasses. These dataclasses are then used to dynamically generate DAGs depending on pipeline type. This setup allows us to implement robust testing, validation, alerts, and documentation for our pipelines. We will also describe the performance improvements we achieved by upgrading to Airflow 2.0.


## Creating Data Pipelines with Elyra, a visual DAG composer and Apache Airflow
[URL](https://airflowsummit.org/sessions/2021/creating-pipelines-elyra/)  
Date: 2021-07-13T04:30:00.000Z
Summary:
This presentation will detail how Elyra creates Jupyter Notebook, Python and R script- based pipelines without having to leave your web browser. The goal of using Elyra is to help construct data pipelines by surfacing concepts and patterns common in pipeline construction into a familiar, easy to navigate interface for Data Scientists and Engineers so they can create pipelines on their own. In Elyra’s Pipeline Editor UI, portions of Apache Airflow’s domain language are surfaced to the user and either made transparent or understandable through the use of tooltips or helpful notes in the proper context during pipeline construction. With these features, Elyra can rapidly prototype data workflows without the need to know or write any pipeline code. Lastly, we will look at what features we have planned on our roadmap for Airflow, including more robust Kubernetes integration and support for runtime specific components/operators. Project Home: https://github.com/elyra-ai/elyra


## Apache Airflow and Ray: Orchestrating ML at Scale
[URL](https://airflowsummit.org/sessions/2021/airflow-ray/)  
Date: 2021-07-13T05:00:00.000Z
Summary:
As the Apache Airflow project grows, we seek both ways to incorporate rising technologies and novel ways to expose them to our users. Ray is one of the fastest-growing distributed computation systems on the market today. In this talk, we will introduce the Ray decorator and Ray backend. These features, built with the help of the Ray maintainers at Anyscale, will allow Data Scientists to natively integrate their distributed pandas, XGBoost, and TensorFlow jobs to their airflow pipelines with a single decorator. By merging the orchestration of Airflow and the distributed computation of Ray, this coordination of technologies opens Airflow users to a whole host of new possibilities when designing their pipelines.


## Event-based Scheduling Based on Airflow
[URL](https://airflowsummit.org/sessions/2021/event-based-scheduling/)  
Date: 2021-07-13T05:30:00.000Z
Summary:
Airflow scheduler uses DAG definitions to monitor the state of tasks in the metadata database, and triggers the task instances whose dependencies have been met. It is based on state of dependencies scheduling. The idea of event based scheduling is to let the operators send events to the scheduler to trigger a scheduling action, such as starting jobs, stopping jobs and restarting jobs. Event based scheduling allows potential support for richer scheduling semantics such as periodic execution and manual trigger at per operator granularity.


## Provision as a Service: Automating data center operations with Airflow at Cloudflare
[URL](https://airflowsummit.org/sessions/2021/provision-as-a-service/)  
Date: 2021-07-13T06:00:00.000Z
Summary:
Cloudflare’s network keeps growing, and that growth doesn’t just come from building new data centers in new cities. We’re also upgrading the capacity of existing data centers by adding newer generations of servers — a process that makes our network safer, faster, and more reliable for our users.
In this talk, I’ll share how we’re leveraging Apache Airflow to build our own Provision-as-a-Service (PraaS) platform and cut by 90% the amount of time our team spent on mundane operational tasks.


## Introducing Viewflow: a framework for writing data models without writing Airflow code
[URL](https://airflowsummit.org/sessions/2021/introducing-viewflow/)  
Date: 2021-07-13T06:30:00.000Z
Summary:
In this talk, we present Viewflow, an open-source Airflow-based framework that allows data scientists to create materialized views in SQL, R, and Python without writing Airflow code.
We will start by explaining what problem does Viewflow solve: writing and maintaining complex Airflow code instead of focusing on data science. Then we will see how Viewflow solves that problem. We will continue by showing how to use VIewflow with several real-world examples. Finally, we will see what the upcoming features of Viewflow are!


## Create Your Custom Secrets Backend for Apache Airflow - A guided tour into Airflow codebase
[URL](https://airflowsummit.org/sessions/2021/create-custom-secrets-backend/)  
Date: 2021-07-13T07:00:00.000Z
Summary:
This talk aims to share how Airflow’s secrets backend works, and how users can create their custom secret backends for their specific use cases & technology stack.


## Airflow and Analytics Engineering - Dos and don'ts
[URL](https://airflowsummit.org/sessions/2021/airflow-and-analytics-engineering/)  
Date: 2021-07-13T16:00:00.000Z
Summary:
Considering that the role of Analytics Engineering has emerged in the last few years within data and analytics teams, it is important for me to highlight what role an Analytics engineer has and how the Dos and Don’ts from my perspective can contribute to a team and boost their day-to-day work with the help of Airflow.


## Robots are your friends - using automation to keep your Airflow operators up to date
[URL](https://airflowsummit.org/sessions/2021/robots-are-your-friends/)  
Date: 2021-07-13T16:00:00.000Z
Summary:
As part of my role at Google, maintaining samples for Cloud Composer, hosted managed Airflow, is crucial. It’s not feasible for me to try out every sample every day to check that it’s working. So, how do I do it?
Automation! While I won’t let the robots touch everything, they let me know when it’s time to pay attention. Here’s how:
Step 0: An update for the operators is released Step 1: A GitHub bot called Renovate Bot opens up a PR to a special requirements file to make this update Step 2: Cloud build runs unit tests to make sure none of my DAGs immediately break Step 3: PR is approved and merged to main Step 4: Cloud Build updates my dev environment Step 5: I look at my DAGs in dev to make sure all is well. If there is a problem, I need to resolve it manually and revert my requirements file. Step 6: I manually update my prod PyPI packages


## SciDAP: Airflow and CWL-powered bioinformatics platform
[URL](https://airflowsummit.org/sessions/2021/scidap/)  
Date: 2021-07-13T16:00:00.000Z
Summary:
Reproducibility is the fundamental principle of a scientific research. This also applies to the computational workflows that are used to process research data. Common Workflow Language (CWL) is a highly formalized way to describe pipelines that was developed to achieve reproducibility and portability of computational analysis. However, there were only few workflow execution platforms that could run CWL pipelines. Here, we present CWL-Airflow – an extension for Airflow to execute CWL pipelines. CWL-Airflow serves as a processing engine for Scientific Data Analysis Platform (SciDAP) – a data analysis platform that makes complex computational workflows both user-friendly and reproducible. In our presentation we are going to explain why we see Airflow as the perfect backend for running scientific workflows, what problems we encountered in extending Airflow to run CWL pipelines and how we solved them. We will also discuss what are the pros and cons of limiting our platform to CWL pipelines and potential applications of CWL-Airflow outside the realm of biology.


## Next-Gen Astronomer Cloud
[URL](https://airflowsummit.org/sessions/2021/next-gen-astronomer-cloud/)  
Date: 2021-07-13T16:30:00.000Z
Summary:
Astronomer founders Ry Walker and Greg Neiheisel will preview the upcoming next-gen Astronomer Cloud product offering.


## Apache Airflow at Wise
[URL](https://airflowsummit.org/sessions/2021/apache-airflow-at-wise/)  
Date: 2021-07-13T17:30:00.000Z
Summary:
Wise (previously TransferWise) is a London-based fin-tech company. We build a better way of sending money internationally.
At Wise we make great use of Airflow. More than 100 data scientists, analysts and engineers use Airflow every day to generate reports, prepare data, (re)train machine learning models and monitor services.
My name is Alexandra, I’m a Machine Learning Engineer at Wise. Our team is responsible for building and maintaining Wise’s Airflow instances. In this presentation I would like to talk about three main things, our current setup, our challenges and our future plans with Airflow. We are currently transitioning from a single centralised Airflow instance into many segregated instances to increase reliability and limit access. We’ve learned a lot throughout this journey and looking to share these learnings with a wider audience.


## Orchestrating ELT with Fivetran and Airflow
[URL](https://airflowsummit.org/sessions/2021/orchestrating-elt-with-fivetran-and-airflow/)  
Date: 2021-07-13T18:00:00.000Z
Summary:
At Fivetran, we are seeing many organizations adopt the Modern Data Stack to suit the breadth of their data needs. However, as incoming data sources begin to scale, it can be hard to manage and maintain the environment, with more time spent repairing and reengineering old data pipelines than building new ones. This talk will introduce a number of new Airflow Providers, including the airflow-provider-fivetran, and discuss some of the benefits and considerations we are seeing data engineers, data analysts, and data scientist experience in doing so.


## Upgrading to Apache Airflow 2
[URL](https://airflowsummit.org/sessions/2021/upgrading-to-apache-airflow-2/)  
Date: 2021-07-13T18:30:00.000Z
Summary:
Airflow 2.0 was a big milestone for the Airflow community. However, companies and enterprises are still facing difficulties in upgrading to 2.0.
In this talk, I would like to focus and highlight the ideal upgrade path and talk about
upgrade_check CLI tool separation of providers registering connections types important 2.0 Airflow configs DB Migration deprecated feature around Airflow Plugins 

## Guaranteeing pipeline SLAs and data quality standards with Databand
[URL](https://airflowsummit.org/sessions/2021/data-quality-standards-databand/)  
Date: 2021-07-14T16:00:00.000Z
Summary:
We’ve all heard the phrase “data is the new oil.” But really imagine a world where this analogy is more real, where problems in the flow of data - delays, low quality, high volatility - could bring down whole economies? When data is the new oil with people and businesses similarly reliant on it, how do you avoid the fires, spills, and crises?
As data products become central to companies’ bottom line, data engineering teams need to create higher standards for the availability, completeness, and fidelity of their data.


## Deep dive in to the Airflow scheduler
[URL](https://airflowsummit.org/sessions/2021/deep-dive-in-to-the-airflow-scheduler/)  
Date: 2021-07-14T16:30:00.000Z
Summary:
The scheduler is the core of Airflow, and it’s a complex beast.
In this session we will go through the scheduler in some detail; how it works; what the communication paths are and what processing is done where.


## Running Big Data Applications in production with Airflow + Firebolt
[URL](https://airflowsummit.org/sessions/2021/big-data-airflow-firebolt/)  
Date: 2021-07-14T17:30:00.000Z
Summary:
In this talk we’ll see some real world examples from Firebolt customers demonstrating how Airflow is used to orchestrate operational data analytics applications with large data volumes, while keeping query latency low.


## Writing Dry Code in Airflow
[URL](https://airflowsummit.org/sessions/2021/writing-dry-code-in-airflow/)  
Date: 2021-07-14T18:00:00.000Z
Summary:
Engineering teams leverage the factory coding pattern to write easy-to-read and repeatable code. In this talk, we’ll outline how data engineering teams can do the same with Airflow by separating DAG declarations from business logic, abstracting task declarations from task dependencies, and creating a code architecture that is simple to understand for new team members. This approach will set analytics teams up for success as team and Airflow DAG sizes grow exponentially.


## Building a robust data pipeline with the dAG stack: dbt, Airflow, Great Expectations
[URL](https://airflowsummit.org/sessions/2021/building-a-robust-data-pipeline-with-the-dag-stack/)  
Date: 2021-07-14T18:30:00.000Z
Summary:
Data quality has become a much discussed topic in the fields of data engineering and data science, and it has become clear that data validation is absolutely crucial to ensuring the reliability of any data products and insights produced by an organization’s data pipelines. This session will outline patterns for combining three popular open source tools in the data ecosystem - dbt, Airflow, and Great Expectations - and use them to build a robust data pipeline with data validation at each critical step.


## An On-Demand Airflow Service for Internet Scale Gameplay Pipelines
[URL](https://airflowsummit.org/sessions/2021/an-on-demand-airflow-service-for-internet-scale-gameplay-pipelines/)  
Date: 2021-07-15T04:00:00.000Z
Summary:
EA Games have very dynamic and federated needs on their data processing pipelines. Many individual studios within EA build and manage the data pipelines for their games iterating rapidly through game development cycles. Developer productivity around orchestrating these pipelines is as critical as providing a robust production quality orchestration service. With these in mind, we re-engineered our Airflow service ground up to cater to our large internal user base (1000s) and internet scale data processing systems (Petabytes of data). This session details the evolution of the use of Airflow at EA Digital Platform from a monolithic multi-tenant instance to an “On-Demand” system where teams and studios create their own dedicated Airflow instance with all the necessary bells-and-whistles required at the click of a button - and allows them to immediately get their data pipelines running. We also elaborate how Airflow is interwoven into a “Self Serve” model for ETL pipelines within our teams with the objective of truely democratizing data across our games.


## Airflow Extensions for Streamlined ETL Backfilling
[URL](https://airflowsummit.org/sessions/2021/airflow-extensions-for-streamlined-etl-backfilling/)  
Date: 2021-07-15T04:30:00.000Z
Summary:
Using Airflow as our scheduling framework, we ETL data generated by tens of millions of transactions every day to build the backbone for our reports, dashboards, and training data for our machine learning models. There are over 500 (and growing) such ingested and aggregated tables owned by multiple teams that contain intricate dependencies between one another. Given this level of complexity, it can become extremely cumbersome to coordinate backfills for any given table, when also taking into account all its downstream dependencies, aggregation intervals, and data availability. This talk will focus on how we customized and extended Airflow at Adyen to streamline our backfilling operations. This allows us to prevent mistakes and enable our product teams to keep launching fast and iterating.


## The new modern data stack - Airbyte, Airflow, DBT
[URL](https://airflowsummit.org/sessions/2021/the-new-modern-data-stack-airbyte-airflow-dbt/)  
Date: 2021-07-15T05:00:00.000Z
Summary:
In this talk, I’ll describe how you can leverage 3 open-source standards - workflow management with Airflow, EL with Airbyte, transformation with DBT - to build your next modern data stack. I’ll explain how to configure your Airflow DAG to trigger Airbyte’s data replication jobs and DBT’s transformation one with a concrete use case.


## Building the Data Science Platform with Airflow @Near
[URL](https://airflowsummit.org/sessions/2021/building-ml-pipelines-with-airflow/)  
Date: 2021-07-15T05:30:00.000Z
Summary:
At Near we work on TBs of Location data with close to real time modelling to generate key consumer insights and estimates for our clients across the globe. We have hundreds of country specific models deployed and managed through airflow to achieve this goal. Some of the workflows that we have deployed our schedule based, some are dynamic and some are trigger based. In this session I would be discussing some of the workflows that are being scheduled and monitored using airflow and the key benefits and also the challenges that we have faced in our production systems.


## Modernize a decade old pipeline with Airflow 2.0
[URL](https://airflowsummit.org/sessions/2021/modernize-a-decade-old-pipeline-with-airflow-2.0)  
Date: 2021-07-15T06:00:00.000Z
Summary:
As a follow up for https://airflowsummit.org/sessions/teaching-old-dag-new-tricks/, in this talk, we would like to share a happy ending story on how Scribd fully migrated its data platform to the cloud and Airflow 2.0.
We will talk about data validation tools and task trigger customizations the team built to smooth out the transition. We will share how we completed the Airflow 2.0 migration started with an unsupported MySQL version and metrics to prove why everyone should perform the upgrade. Lastly, we will discuss how large scale backfills (10 years worth of run) are managed and automated at Scribd.


## Building the AirflowEventStream
[URL](https://airflowsummit.org/sessions/2021/building-the-airfloweventstream/)  
Date: 2021-07-15T07:00:00.000Z
Summary:
Or how to keep our traditional java application up-to-date on everything big data.
At Adyen we process tens of millions of transactions a day, a number that rises every day. This means that generating reports, training machine learning models or any other operation that requires a bird’s eye view on weeks or months of data requires the use of Big Data technologies.
We recently migrated to Airflow for scheduling all batch operations on our on-premise Big Data cluster. Some of these operations require input from our merchants or our support team. Merchants can for instance subscribe to reports, choose their preferred time zone, and even specify which columns they want included. After generating the reports, these reports then need to become available in our customer portal.


## Dynamic Security Roles in Airflow for Multi-Tenancy
[URL](https://airflowsummit.org/sessions/2021/dynamic-security-roles-in-airflow-for-multi-tenancy/)  
Date: 2021-07-15T16:00:00.000Z
Summary:
Multi-tenant Airflow instances can help save costs for an organization. This talk will walk through how we dynamically assigned roles to users based on groups in Active Directory so that teams would have access to DAGs they created in the UI on our multi-tenant Airflow instance. We created our own custom AirflowSecurityManager class in order to achieve this that ultimately ties LDAP and RBAC together.


## Apache Airflow 2.0 on Amazon MWAA
[URL](https://airflowsummit.org/sessions/2021/airflow-aws/)  
Date: 2021-07-15T16:30:00.000Z
Summary:
In this session we will discuss Amazon Managed Workflows for Apache Airflow (MWAA), how Apache Airflow (and specifically version 2.0) is implemented in the service, best practices for deployment and operations, and the Amazon MWAA team’s commitment to open source usage and contributions.


## Clearing Airflow obstructions
[URL](https://airflowsummit.org/sessions/2021/clearing-airflow-obstructions/)  
Date: 2021-07-15T17:30:00.000Z
Summary:
Apache Airflow aims to speed the development of workflows, but developers are always ready to add bugs here and there.
This talk illustrates a few pitfalls faced while developing workflows at the BBC to build machine learning models. The objective is to share some lessons learned and, hopefully, save others time.
Some of the topics covered, with code examples:
Tasks unsuitable to be run from within Airflow executors Plugins misusage Inconsistency while using an operator (Mis)configuration What to avoid during a workflow deployment Consequences of non-idempotent tasks 

## Building Providers & DAGs in the Airflow Ecosystem
[URL](https://airflowsummit.org/sessions/2021/learn-how-to-build-your-own-airflow-provider/)  
Date: 2021-07-15T18:30:00.000Z
Summary:
Learn how to use Airflow’s robust ecosystem of providers to construct secure, high-quality DAGs.


## Reverse ETL on Airflow
[URL](https://airflowsummit.org/sessions/2021/reverse-etl-on-airflow/)  
Date: 2021-07-15T19:00:00.000Z
Summary:
At Snowflake you can imagine we do a lot of data pipelines and tables curating metrics metrics for all parts of the business. These are the lifeline of Snowflake’s business decisions. We also have a lot of source systems that display and make these metrics accessible to end users. So what happens when your data model does not match your system? For example your bookings numbers in salesforce do not match your data model that curates bookings metrics. At snowflake we continued to run into this problem over and over again.


## Productionizing ML Pipelines with Airflow, Kedro, and Great Expectations
[URL](https://airflowsummit.org/sessions/2021/productionizing-ml-pipelines-with-airflow-kedro-great-expectations/)  
Date: 2021-07-16T04:00:00.000Z
Summary:
Machine Learning models can add value and insight to many projects, but they can be challenging to put into production due to problems like lack of reproducibility, difficulty maintaining integrations, and sneaky data quality issues. Kedro, a framework for creating reproducible, maintainable, and modular data science code, and Great Expectations, a framework for data validations, are two great open-source Python tools that can address some of these problems. Both integrate seamlessly with Airflow for flexible and powerful ML pipeline orchestration. In this talk we’ll discuss how you can leverage existing Airflow provider packages to integrate these tools to create sustainable, production-ready ML models.


## Data Lineage with Apache Airflow using OpenLineage
[URL](https://airflowsummit.org/sessions/2021/data-lineage-with-apache-airflow-using-openlineage/)  
Date: 2021-07-16T04:30:00.000Z
Summary:
If you manage a lot of data, and you’re attending this summit, you likely rely on Apache Airflow to do a lot of the heavy lifting. Like any powerful tool, Apache Airflow allows you to accomplish what you couldn’t before… but also creates new challenges. As DAGs pile up, complexity layers on top of complexity and it becomes hard to grasp how a failed or delayed DAG will affect everything downstream. In this session we will provide a crash course on OpenLineage, an open platform for metadata management and data lineage analysis. We’ll show how capturing metadata with OpenLineage can help you maintain inter-DAG dependencies, capture data on historical runs, and minimize data quality issues.


## Drift Bio: The Future of Microbial Genomics with Apache Airflow
[URL](https://airflowsummit.org/sessions/2021/drift-bio--futuremicrobial-genomics-with-apache-airflow/)  
Date: 2021-07-16T05:30:00.000Z
Summary:
In recent years, the bioinformatics world has seen an explosion in genomic analysis as gene sequencing technologies have become exponentially cheaper. Tests that previously would have cost tens of thousands of dollars will soon run at pennies per sequence. This glut of data has exposed a notable bottleneck in the current suite of technologies available to bioinformaticians. At Drift Biotechnologies, we use Apache Airflow to transition traditionally on-premise large scale data and deep learning workflows for bioinformatics to the cloud, with an emphasis on workflows and data from next generation sequencing technologies.


## Usability Improvements: Debugging & Inspection Tooling
[URL](https://airflowsummit.org/sessions/2021/usability-improvements-debugging-inspection-tooling/)  
Date: 2021-07-16T06:00:00.000Z
Summary:
The two most common user questions at Pinterest are: 1) why is my workflow running so long? 2) why did my workflow fail - is it my issue, or a platform issue? As with any big data organization, the workflow platform is just the orchestrator but the “real” work is done on another layer, managed by another platform. There can be plenty of these, and the challenges of figuring out the root cause of an issue can be mundane and time consuming. At Pinterest, we set out to provide additional tooling in our Airflow webserver to make it a quicker inspection process and provide smart tips such as increased runtime analysis, bottleneck identifying, rca, and an easy way for backfilling. We explore deeper the tooling provided to reduce the admin load, and empower our users.


## Autoscaling in Airflow - Lessons learned
[URL](https://airflowsummit.org/sessions/2021/autoscaling-airflow/)  
Date: 2021-07-16T07:00:00.000Z
Summary:
Autoscaling in Airflow - what we learnt based on Cloud Composer case. We would like to present how we approach the autoscaling problem for Airflow running in Kubernetes in Cloud Composer: how we calculate our autoscaling metric, what problem we had for scaling down and how did we solve it. Also we share an ideas on what and how we could improve the current solution


## Building a Scalable & Isolated Architecture for Preprocessing Medical Records
[URL](https://airflowsummit.org/sessions/2021/building-scalable-isolated-architecture-for-preprocessing-medical-records/)  
Date: 2021-07-16T16:00:00.000Z
Summary:
After performing several experiments with Airflow, we reached the best architectural design for processing text medical records in scale. Our hybrid solution uses Kubernetes, Apache Airflow, Apache Livy, and Apache cTAKES. Using Kubernetes’ containers has the benefit of having a consistent, portable, and isolated environment for each component of the pipeline. With Apache Livy, you can run tasks in a Spark Cluster at scale. Additionally, Apache cTAKES helps with the extraction of information from electronic medical records clinical free-text by using natural language processing techniques to identify codable entities, temporal events, properties, and relations.


## Advanced Superset for Engineers (API’s, Version Controlled Dashboards, & more)
[URL](https://airflowsummit.org/sessions/2021/advanced-superset/)  
Date: 2021-07-16T16:30:00.000Z
Summary:
Apache Superset is a modern, open-source data exploration & visualization platform originally created by Maxime Beauchemin. In this talk, I will showcase advanced technical Superset features like the rich Superset API, how to version control dashboards using Github, embedding Superset charts in other applications, and more. This talk will be technical and hands-on, and I will share all code examples I use so you can play with them yourself afterwards!


## Operating contexts: patterns around defining how a DAG should behave in dev, staging, prod & beyond
[URL](https://airflowsummit.org/sessions/2021/operating-contexts-patterns/)  
Date: 2021-07-16T17:00:00.000Z
Summary:
As people define and publish a DAG, it can be really useful to make it clear how this DAG should behave under different “operating contexts”. Common operating contexts may match your different environments (dev / staging / prod) and/or match your operating needs (quick run, full backfill, test run, …).
Over the years, patterns have emerged around workflow authors, teams and organizations, and little has been shared as to how to approach this. In this talk, we’ll talk about what an “operating context” is, why it’s useful, and describe common patterns and best practices around this topic.


## Airflow loves Kubernetes
[URL](https://airflowsummit.org/sessions/2021/airflow-loves-kubernetes/)  
Date: 2021-07-16T17:30:00.000Z
Summary:
In this talk Jarek and Kaxil will talk about official, community support for running Airflow in the Kubernetes environment.
The full support for Kubernetes deployments was developed by the community for quite a while and in the past users of Airflow had to rely on 3rd-party images and helm-charts to run Airflow on Kubernetes. Over the last year community members made an enormous effort to provide robust, simple and versatile support for those deployments that would respond to all kinds of Airflow users. Starting from official container image, through quick-start docker-compose configuration, culminating in April with release of the official Helm Chart for Airflow.


## Customizing Xcom to enhance data sharing between tasks
[URL](https://airflowsummit.org/sessions/2021/customizing-xcom-to-enhance-data-sharing-between-tasks/)  
Date: 2021-07-16T18:00:00.000Z
Summary:
In Apache Airflow, Xcom is the default mechanism for passing data between tasks in a DAG. In practice, this has been restricted to small data elements, since the Xcom data is persisted in the Airflow metadatabase and is constrained by database and performance limitations.
With the new TaskFlow API introduced in Airflow 2.0, it is seamless to pass data between tasks and the use of Xcom is invisible. However, the ability to pass data is restricted to a relatively small set of data types which can be natively converted in JSON.


## MWAA: Design Choices and Road Ahead
[URL](https://airflowsummit.org/sessions/2021/mwaa/)  
Date: 2021-07-16T19:00:00.000Z
Summary:
An informal and fun chat about the journey that we took and the decisions that we made in building Amazon Managed Workflows for Apache Airflow. We will talk about
Our first tryst with understanding Airflow Talking to Amazon Data Engineers and how they ran workflows at scale Key design decisions and the reasons behind them Road ahead, and what we dream about for future of Apache Airflow. Open-Source tenets and commitment from the team We will leave time at the end for a short AMA/Questions.


## Dynamic Dags -- The New Horizon
[URL](https://airflowsummit.org/sessions/2022/dynamic-dags-new-horizon/)  
Date: 2022-05-23T15:00:00.000Z
Summary:
In Airflow 2.3 the ability to change the number of tasks dynamically opens up some exciting new ways of building DAGs and lets us create new patterns that just weren’t possible before.
In this session I will cover a little bit about AIP-42 and the interface for Dynamic Task Mapping, and cover some common use cases and patterns.


## How to eliminate Data Downtime & start trusting your data
[URL](https://airflowsummit.org/sessions/2022/how-to-eliminate-data-downtime/)  
Date: 2022-05-23T16:00:00.000Z
Summary:
Broken data is costly, time-consuming, and nowadays, an all-too-common reality for even the most advanced data teams. In this talk, I’ll introduce this problem, called “data downtime” — periods of time when data is partial, erroneous, missing or otherwise inaccurate — and discuss how to eliminate it in your data ecosystem with end-to-end data observability. Drawing corollaries to application observability in software engineering, data observability is a critical component of the modern DataOps workflow and the key to ensuring data trust at scale. I’ll share why data observability matters when it comes to building a better data quality strategy and highlight tactics you can use to address it today.


## Manage Dags at scale (Dags versioning & package management)
[URL](https://airflowsummit.org/sessions/2022/manage-dags-at-scale/)  
Date: 2022-05-23T16:30:00.000Z
Summary:
This talk is all about how we at Jagex manage DAGs at scale. Focusing of following challenges faced and how we resolved them;
Keeping track of airflow state
Keeping track of each DAG state
DAGs as git submodules
Updating airflow with new dags
How to seamlessly automate airflow deployment
How to avoid package dependency conflicts


## Using Apache Airflow to orchestrate workflows across hybrid environments
[URL](https://airflowsummit.org/sessions/2022/orchestrate-hybrid/)  
Date: 2022-05-23T17:30:00.000Z
Summary:
According to analysts, 87 percent of enterprises have already adopted hybrid cloud strategies (https://www.flexera.com/blog/industry-trends/trend-of-cloud-computing-2020/). Customers have many reasons why they need to support hybrid environments, from maximising the value from heritage systems, to meeting local compliance and data processing regulations. As they build their data pipelines, they increasingly need to be able to orchestrate those across on-premesis and cloud environments. In this session, I will share how you can leverage Apache Airflow to orchestrate a workflow using data sources inside and outside the cloud.


## Choosing Apache Airflow over other Proprietary Tools for your Orchestration needs
[URL](https://airflowsummit.org/sessions/2022/choosing-apache-airflow/)  
Date: 2022-05-23T20:00:00.000Z
Summary:
Organizations need to effectively manage large volumes of complex, business-critical workloads across multiple applications and platforms. Choosing the right workflow orchestration tool is important as it can help teams effectively automate the configuration, coordination, integration, and data management processes on several applications and systems. Currently there are a lot of tools (both open sourced and proprietary) available for orchestrating tasks and data workflows with automation features. Each of them claim to focus on ensuring a centralized, repeatable, reproducible, and efficient workflows coordination. Choosing one among them is an arduous task as it requires an in-depth understanding of the capabilities that these tools have to offer that translate to your specific engineering needs.


## Airflow at Shopify: Keeping Users Happy while Running Airflow at Scale
[URL](https://airflowsummit.org/sessions/2022/airflow-at-shopify/)  
Date: 2022-05-23T21:00:00.000Z
Summary:
Two years after starting our Airflow adoption, we’re running over 10,000 DAGs in production. On this journey we’ve learned a lot about Airflow management and stewardship and developed some unique tools to help us scale.
We’re excited to share our experience and some of the lessons we’ve picked up along the way. In this talk we’ll cover:
The history of Airflow at Shopify Our infrastructure and architecture Custom tools and procedures we’ve adopted to keep Airflow running smoothly and our users happy 

## What is data lineage and why should I care?
[URL](https://airflowsummit.org/sessions/2022/what-is-data-lineage-and-why-should-i-care/)  
Date: 2022-05-23T21:30:00.000Z
Summary:
If a job fails, how can you learn about downstream datasets that have become out-of-date? Can you be confident that jobs are consuming fresh, high-quality data from their upstream sources? How might you predict the impact of a planned change on distant corners of the pipeline?
These questions become easier once you have a complete understanding of data lineage, the complex set of relationships between all of your jobs and datasets. In this talk, Ross Turk from Datakin will provide a quick introduction to the core concepts behind data lineage and an overview of common architectural approaches.


## Vega: Unifying Machine Learning Workflows at Credit Karma using Apache Airflow
[URL](https://airflowsummit.org/sessions/2022/vega-unifying-ml-workflows-credit-karma/)  
Date: 2022-05-24T03:00:00.000Z
Summary:
At Credit Karma, we enable financial progress for more than 100 million of our members by recommending them personalized financial products when they interact with our application. In this talk we are introducing our machine learning platform to build interactive and production model-building workflows to serve relevant financial products to Credit Karma users.
Vega, Credit Karma’s Machine Learning Platform, has 3 major components: 1) QueryProcessor for feature and training data generation, backed by Google BigQuery, 2) PipelineProcessor for feature transformations, offline scoring and model-analysis, backed by Apache Beam 3) ModelProcessor for running Tensorflow and Scikit models, backed by Google AI Platform, which provides data scientists the flexibility to explore different kinds of machine learning or deep learning models, ranging from gradient boosted trees to neural network with complex structures


## Preventative Metadata: Building for data reliability with DataHub, Airflow, & Great Expectations
[URL](https://airflowsummit.org/sessions/2022/preventative-metadata/)  
Date: 2022-05-24T04:00:00.000Z
Summary:
Recently there has been much discussion around data monitoring, particularly in regards to reducing time to mitigate data quality problems once they’ve been detected. The problem with reactive or periodic monitoring as the de-facto standard for maintaining data quality is that it’s expensive. By the time a data problem has been identified, it’s effects may have been amplified across a myriad of downstream consumers, leaving you (a data engineer) with a big mess to clean-up.


## Airflow at high scale for Autonomous Driving
[URL](https://airflowsummit.org/sessions/2022/airflow-at-high-scale-for-autonomous-driving/)  
Date: 2022-05-24T05:00:00.000Z
Summary:
This talk highlights a large-scale use case of Airflow to orchestrate workflows for an Autonomous Driving project based in Germany.
To support our customer in the aim of producing their first Level-3 Autonomous Driving vehicle in Germany, we are utilising Airflow as a state-of-the-art tool to orchestrate workloads running on a large-scale HPC platform.
In this talk, we will describe our Airflow setup deployed on OpenShift that is capable of running thousands of tasks in parallel and contains various custom improvements optimised for our use case. We will in detail discuss how we achieved to integrate multiple components with airflow such as a PostgreSQL database, a highly available RabbitMQ message broker and a fully integrated IAM solution. In particular, we will describe bottlenecks which we have encountered in our journey towards scaling up airflow, and how we mitigate them. We will also detail on the custom improvements we implemented on both airflow code base and deployment setup, such as an enhanced airflow logging framework, improvements on the Spark submit operator, and a feature to re-deploy airflow without business downtime.


## Happy DAGs + Happy Teammates: How a little CI/CD can go a long way
[URL](https://airflowsummit.org/sessions/2022/happy-dags-happy-teammates/)  
Date: 2022-05-24T15:00:00.000Z
Summary:
With a small amount of Cloud Build automation and the use of GitHub version control, your Airflow DAGs will always be tested and in sync no matter who is working on them. Leah will walk you through a sample CICD workflow for keeping your Airflow DAGs tested and in sync between environments and teammates.


## A look under the hood of the Airflow logging subsystem
[URL](https://airflowsummit.org/sessions/2022/under-the-hood-of-the-airflow-logging-subsystem/)  
Date: 2022-05-24T15:30:00.000Z
Summary:
The task logging subsystem is one of most flexible, yet complex and misunderstood components of Airflow.
In this talk, we will take a look at the various task log handlers that are part of the core Airflow distribution, and dig a bit deeper in the interfaces they implement and discuss how those can be used to roll your own logging implementation.


## Keep Calm & Query On: Debugging Broken Data Pipelines with Airflow
[URL](https://airflowsummit.org/sessions/2022/keep-calm-query-on/)  
Date: 2022-05-24T16:00:00.000Z
Summary:
“Why is my data missing?” “Why didn’t my Airflow job run?” “What happened to this report?”
If you’ve been on the receiving end of any of these questions, you’re not alone. As data pipelines become increasingly complex and companies ingest more and more data, data engineers are on the hook for troubleshooting where, why, and how data quality issues occur, and most importantly, fixing them so systems can get up and running again. In this talk, Francisco Alberini, Monte Carlo’s first product hire, discusses the three primary factors that contribute to data quality issues and how data teams can leverage Airflow, dbt, and other solutions in their arsenal to conduct root cause analysis on their data pipelines.


## How to Achieve Reliable Data in your Airflow Pipelines with Databand
[URL](https://airflowsummit.org/sessions/2022/how-to-achieve-reliable-data-in-your-airflow-pipelines-with-databand/)  
Date: 2022-05-24T17:00:00.000Z
Summary:
Have data quality issues?
What about reliability problems? You may be hearing a lot of these terms, along with many others, that describe issues you face with your data. What’s the difference, which are you suffering from, and how do you tackle both?
Knowing that your Airflow DAGs are green is not enough. It’s time to focus on data reliability and quality measurements to build trust in your data platform.


## Future of the Airflow UI
[URL](https://airflowsummit.org/sessions/2022/future-airflow-ui/)  
Date: 2022-05-24T17:30:00.000Z
Summary:
Sneak peek at the future of the Airflow UI. In Airflow 2.3 with the Tree -> Grid view changes, we began to swap out parts of the Flask app with React. This was one step towards AIP-38, to build a fully modern UI for Airflow. Come check out what is in store after Grid view in the current UI.
Discuss the possibilities to rethink Airflow with a brand new UI down the line. Such as:


## Leveraging Open Source Projects For Personal Development
[URL](https://airflowsummit.org/sessions/2022/leveraging-open-source-projects-for-personal-development/)  
Date: 2022-05-24T18:00:00.000Z
Summary:
Have you ever wondered what is next after learning the basics of software development, how you can improve your programming skills and gain more experience?
These questions trouble a lot of people new to software development. They are not aware that they can leverage open-source projects to build their careers and land their dream job.
In this session, I will share how you can leverage open-source projects to improve your skills, the challenges you would likely encounter, and how to overcome them and become a successful software engineer.


## All About Deferrables
[URL](https://airflowsummit.org/sessions/2022/all-about-deferrables/)  
Date: 2022-05-24T20:00:00.000Z
Summary:
Airflow 2.2 introduced Deferrable Tasks (sometimes called “async operators”), a new mechanism to efficiently run tasks that depend on external activity.
But when should you use them, how do they work, what do you need to do to make Operators support them, and what else could we do in Airflow with this model?


## git push your data stack with Airbyte, Airflow and dbt
[URL](https://airflowsummit.org/sessions/2022/git-push-your-data-stack-with-airbyte-airflow-and-dbt/)  
Date: 2022-05-24T20:30:00.000Z
Summary:
The use of version control and continuous deployment in a data pipeline is one of the biggest features unlocked by the modern data stack. In this talk, I’ll demonstrate how to use Airbyte to pull data into your data warehouse, dbt to generate insights from your data, and Airflow to orchestrate every step of the pipeline. The complete project will be managed by version control and continuously deployed by Github. This talk will share how to achieve a more secure, scalable, and manageable workflow for your data projects.


## Managing Apache Airflow at Scale
[URL](https://airflowsummit.org/sessions/2022/managing-apache-airflow-at-scale/)  
Date: 2022-05-24T21:00:00.000Z
Summary:
In this session we’ll be discussing the considerations and challenges when running Apache Airflow at scale.
We’ll start by defining what it means to run Airflow at scale. Then we’ll dive deep into understanding limitations of the Airflow architecture, Scheduler processes, and configuration options.
We’ll then define scaling workloads via containers and leveraging pools and priority, followed by scaling DAGs via dDynamic DAGs/DAG factories, CI/CD, and DAG access control.
Finally we’ll get into managing Multiple Airflow Environments, how to split up workloads, and provide central governance for Airflow environment creation and monitoring with an example of Distributing workloads across environments.


## Wisdoms learnt when contributing to Apache Airflow
[URL](https://airflowsummit.org/sessions/2022/wisdoms-learnt-when-contributing/)  
Date: 2022-05-25T04:00:00.000Z
Summary:
In this talk, I am going to share things that I learned while contributing to Apache Airflow. I am an Outreachy Intern for Apache Airflow. I made my first contribution to Open Source in the Apache Airflow project. I will also add a short description about myself and my experience working in Software Engineering and how i needed help in contributing to open source and ended up as an Intern for Outreachy. I also like to share about my first contribution towards Apache Airflow in its doc and how much confidence it gave me to continue contributing to it.


## Automatic Speech Recognition at Scale Using Tensorflow, Kubernetes and Airflow
[URL](https://airflowsummit.org/sessions/2022/automatic-speech-recognition-at-scale/)  
Date: 2022-05-25T05:30:00.000Z
Summary:
Automatic Speech Recognition is quite a compute intensive task, which depends on complex Deep Learning models. To do this at scale, we leveraged the power of Tensorflow, Kubernetes and Airflow.
In this session, you will learn about our journey to tackle this problem, main challenges, and how Airflow made it possible to create a solution that is powerful, yet simple and flexible.


## Multitenancy is coming
[URL](https://airflowsummit.org/sessions/2022/multitenancy-is-coming/)  
Date: 2022-05-25T06:00:00.000Z
Summary:
This session is about the state and future plans of the multi-tenancy feature of Airflow.
Airflow has traditionally been single-tenant product. Mutliple instances could be bound together to provide a multi-tenant implementation and when using a modern infrastructure - Kubernetes - you could even reuse resources between those - but it was not a true “multi-tenant” solution.
But Airflow becomes more of a platform now and the needs for multi-tenancy as a feature of the platform are highly expected by a number of users.


## Airflow in the Cloud: Lessons from the Field
[URL](https://airflowsummit.org/sessions/2022/airflow-in-the-cloud-lessons-from-the-field/)  
Date: 2022-05-25T15:00:00.000Z
Summary:
Airflow users love to run Airflow in public clouds and on distributed infrastructures like Kubernetes. Running Airflow environments is easier than ever - community offers Helm-based installation for self-managed Airflow and there are many offerings of Airflow-based managed services.
Commoditization of Airflow and broader Airflow user base brings new challenges. This talk presents observations of the Airflow service provider delivering “Airflow as a Service’’ to cloud users (very technical, less technical and not technical at all). Information presented during this talk will be directed to the Apache Airflow committers and contributors with the hope that one can influence Airflow’s future roadmap so that Apache Airflow becomes easy to use.


## OpenLineage & Airflow - data lineage has never been easier
[URL](https://airflowsummit.org/sessions/2022/openlineage/)  
Date: 2022-05-25T16:00:00.000Z
Summary:
OpenLineage is an open standard for metadata and lineage collection designed to instrument jobs as they are running. The standard has become remarkably adept at understanding the lifecycle of data within an organization. Additionally, Airflow lets you make use of OpenLineage with a convenient integration. Gathering data lineage has never been easier. In this talk, we’ll provide an update-to-date report on OpenLineage features and the Airflow integration – essential information for data governance architects & engineers.


## How DAG Became a Test - Airflow System Tests Redefined
[URL](https://airflowsummit.org/sessions/2022/how-dag-became-a-test/)  
Date: 2022-05-25T16:30:00.000Z
Summary:
Nothing is perfect, but it doesn’t mean we shouldn’t seek perfection. After some time spent with Airflow system tests, we have recognized numerous places in which we can make significant improvements. We decided to rediscover them. The new design started with the establishment of goals.
Tests need to:
be easy to write, read, run and maintain, be as close as possible to how Airflow runs in practice, be fast, reliable and verifiable, assure high quality of Airflow Operators. With these principles in mind, we prepared an Airflow Improvement Proposal (AIP-47) and after it’s confirmed, we started the implementation. The results of our work were better than we expected when we started this initiative.


## Running +150 production Airflow on Kubernetes, is that HARD ?
[URL](https://airflowsummit.org/sessions/2022/running-production-airflow-kubernetes/)  
Date: 2022-05-25T17:00:00.000Z
Summary:
This talk will cover the challenges we can face managing a large number of Airflow instances on private environment.
Monitoring and metrics layers for production environment. Collecting and customizing logs. Resource consumption and green IT. Providing support for users and shared responsibility. Pain points 

## On-Demand DAG through the REST API
[URL](https://airflowsummit.org/sessions/2022/on-demand-dag/)  
Date: 2022-05-25T20:00:00.000Z
Summary:
In this talk we want to present how Airbnb extends the REST api to support on-demand workload. A DAG object is created from a local environment like Jupyter notebook, serialized into binary and transported to the API. The API persists the DAG object into the meta DB and Airflow scheduler and worker are extended to process this new kind of DAG.


## Airflow at Pinterest
[URL](https://airflowsummit.org/sessions/2022/airflow-at-pinterest/)  
Date: 2022-05-25T21:00:00.000Z
Summary:
Pinterest has been part of the Airflow community for two years and has worked on many custom solutions to address usability, scalability, and efficiency constraints. This session is to discuss how Pinterest has further expanded on those previous solutions. We will discuss how we work to further reduce system latencies, improve user development through added search features, support for cross cluster operations, and improved debuggability tooling, and system level efficiency improvements to auto retry failed tasks that meet certain criteria.


## Using the Fivetran Airflow Provider
[URL](https://airflowsummit.org/sessions/2022/using-fivetran-provider/)  
Date: 2022-05-25T21:30:00.000Z
Summary:
Fivetran’s Airflow provider allows Recharge to manage our connector syncs alongside our other DAGs orchestrating related components of our core data pipelines. The provider has enabled increased flexibility on sync schedules, custom alerting, and quicker response times to failures.


## The tale of a startup's data journey and its growing need for orchestration
[URL](https://airflowsummit.org/sessions/2022/startup-data-journey/)  
Date: 2022-05-25T22:00:00.000Z
Summary:
This talk tells the story of how we have approached data and analytics as a startup at Preset and how the need for a data orchestrator grew over time. Our stack is (loosely) Fivetran/Segment/dbt/BigQuery/Hightouch, and we finally got to a place where we suffer quite a bit from not having an orchestrator and are bringing in Airflow to address our orchestration needs. This talk is about how startups approach solving data challenges, the shifting role of the orchestrator in the modern data stack, and the growing need for an orchestrator as your data platform becomes more complex.


## Introducing Astro Python SDK: The next generation of DAG authoring
[URL](https://airflowsummit.org/sessions/2022/introducing-astro-dag-authoring/)  
Date: 2022-05-26T03:00:00.000Z
Summary:
Imagine if you could chain together SQL models using nothing but python, write functions that treat Snowflake tables like dataframes and dataframes like SQL tables. Imagine if you could write a SQL airflow DAG using only python or without using any python at all.
With Astro SDK, we at Astronomer have gone back to the drawing board around fundamental questions of what DAG writing could look like. Our goal is to empower Data Engineers, Data Scientists, and even the Business Analysts to write Airflow DAGs with code that reflects the data movement, instead of the system configuration. Astro will allow each group to focus on producing value in their respective fields with minimal knowledge of Airflow and high amounts of flexibility between SQL or python-based systems.


## Automating Airflow Backfills with Marquez
[URL](https://airflowsummit.org/sessions/2022/automating-airflow-backfills-with-marquez/)  
Date: 2022-05-26T04:00:00.000Z
Summary:
As a data engineer, backfilling data is an important part of your day-to-day work. But, backfilling interdependent DAGs is time-consuming and often associated with an unpleasant experience. For example, let’s say you were tasked with backfilling a few months worth of data. You’re given the start and end date for the backfill that will be used to run an ad-hoc backfilling script that you have painstakingly crafted locally on your machine. As you sip your morning coffee, you kick off the backfilling script, hoping it’ll work, and think to yourself, there must be a better way. Yes, there is, and collecting DAG lineage metadata would be a great start!


## Airflow & Zeppelin: Better together
[URL](https://airflowsummit.org/sessions/2022/airflow-zeppelin/)  
Date: 2022-05-26T05:00:00.000Z
Summary:
Airflow is the almost de-facto standard job orchestration tool that is used in the production stage. But moving your job from the development stage in other tools to the production stage in Airflow is usually a big pain for lots of users. A major reason is due to the environment inconsistency between the development environment and the production environment. Apache Zeppelin is a web-based notebook that is integrated seamlessly with lots of popular big data engines, such as Spark, Flink, Hive, Presto and etc. So it is very suitable for the development stage.


## Airflow / Kubernetes: Running on and using k8s
[URL](https://airflowsummit.org/sessions/2022/airflow-k8s/)  
Date: 2022-05-26T05:30:00.000Z
Summary:
Apache Airflow and Kubernetes work well together. Not only does Airflow have native support for running tasks on Kubernetes, there is also an official helm chart that makes it easy to run Airflow itself on Kubernetes!
Confused on the differences between KubernetesExecutor and KubernetesPodOperator? What about CeleryKubernetesExecutor? Or the new LocalKubernetesExecutor? After this talk you will understand how they all fit in the ecosystem.
We will talk about the ways you can run Airflow on Kubernetes, run tasks on Kubernetes, or do both. We will also cover things you may want to consider doing to have a reliable Airflow instance.


## Skip tasks to make your debugging easy
[URL](https://airflowsummit.org/sessions/2022/skip-tasks-to-make-your-debugging-easy/)  
Date: 2022-05-26T05:40:00.000Z
Summary:
In Apple, we are building a self-serve data platform based on Airflow. Self-serve means users can create, deploy and run their DAGs freely. With provided logs and metrics, users are able to test or troubleshot DAGs on their own. Today, a common use case is, users want to test one or a few tasks in their DAG. However, when they trigger the DAG, all tasks instead of just the ones people are interested will run. To save time and resources, lots of users choose to manually mark complete for each tasks to skip. Can we do better than that? Is there an easy-peasy way to skip tasks?


## The SLAyer your Data Pipeline Needs
[URL](https://airflowsummit.org/sessions/2022/the-slayer-your-data-pipeline-needs/)  
Date: 2022-05-26T15:00:00.000Z
Summary:
Airflow has an inherent SLA alert mechanism. When the scheduler sees such an SLA miss for some task, it sends an alert by email. The problem is, that this email is nice, but we can’t really know when each task is eventually successful. Moreover, even if there is such an email upon success following an SLA miss, it does not give us a good view of the current status at any given time.


## Managing Multiple ML Models For Multiple Clients : Steps For Scaling Up
[URL](https://airflowsummit.org/sessions/2022/managing-multiple-ml-models-for-multiple-clients/)  
Date: 2022-05-26T15:30:00.000Z
Summary:
For most ML-based SaaS companies, the need to fulfill each customer’s KPI will usually be addressed by matching a dedicated model. Along with the benefits of optimizing the model’s performance, a model per customer solution carries a heavy production complexity with it. In this manner, incorporating up-to-date data as well as new features and capabilities as part of a model’s retraining process can become a major production bottleneck. In this talk, we will see how Riskified scaled up modeling operations based on MLOps ideas, and focus on how we used Airflow as our ML pipeline orchestrator. We will dive into how we wrap Airflow as an internal service, the goals we started with, the obstacles along the way and finally - how we solved them. You will receive tools for how to set up your own Airflow-based continuous training ML pipeline, and how we adjusted it such that ML engineers and data scientists would be able to collaborate and work in parallel using the same pipeline.


## Lets use Airflow differently: let's talk load tests
[URL](https://airflowsummit.org/sessions/2022/lets-use-airflow-differently-load-tests/)  
Date: 2022-05-26T16:00:00.000Z
Summary:
Numeric results with bulletproof confidence: this is what companies actually sell when promoting their machine learning product. Yet this seems out of reach when the product is both generic and complex, with much of the inner calculations hidden from the end user.
So how can code improvements or changes in core component performance be tested at scale? Implementing API and Load Tests is time-consuming, but thorough: defining parameters, building infrastructure and debugging. The bugs may be real, but they can also be a result of poor infrastructure implementation (who is testing the testers?).


## How to Deploy Airflow From Dev to Prod Like A BOSS
[URL](https://airflowsummit.org/sessions/2022/how-to-deploy-airflow-from-dev-to-prod-like-a-boss/)  
Date: 2022-05-26T17:00:00.000Z
Summary:
Managing Airflow in large-scale environments is tough. You know this, and I know this.
But, what if you had a guide to make development, testing, and production lifecycles more manageable?
In this presentation, I will share how we manage Airflow for large-scale environments with friendly deployments at every step.
After attending the session, Airflow engineers will:
Understand the advantages of each kind of deployment Know the differences between Deployment and Airflow Executor Deploy how to incorporate all kinds of deployments for their day-to-day needs 

## Well-Architected Workflows in Apache Airflow
[URL](https://airflowsummit.org/sessions/2022/well-architected-workflows/)  
Date: 2022-05-26T20:00:00.000Z
Summary:
Resilient systems have the capability to recover when stressed by load, bugs in the workflow, and failure of any task. Reliability of the infrastructure or platform is not sufficient to run workflows reliably. It is critical to bring in resiliency practices during the design and build phase of the workflow to improve reliability, performance and operational aspects of the workflow.
In this session, We will go through
Architecture of the Airflow through the lens of reliability Idempotency Designing for failures Applying back pressure Best practices What we do not cover: Infrastructure/Platform/Product reliability


## Modern Data Orchestration managed by Astronomer
[URL](https://airflowsummit.org/sessions/2022/astronomer/)  
Date: 2022-05-26T20:30:00.000Z
Summary:
At Astronomer we have been longtime supporters and contributors to open source Apache Airflow. In this session we will present Astronomer’s latest journey, Astro, our cloud-native managed service that simplifies data orchestration and reduces operational overhead. We will also discuss the increasing importance of data orchestration in modern enterprise data platforms, industry trends, and practical problems that arise in the ever expanding heterogeneous environments.


## Airflow extensions for governing a self-serviced data mesh
[URL](https://airflowsummit.org/sessions/2022/airflow-extensions-governance/)  
Date: 2022-05-26T21:00:00.000Z
Summary:
While many companies set up isolated data teams, Adyen is a strong believer of the data mesh approach, with all our data living in a central place. While our tooling teams provide and operate the on-premise cluster, the product teams are able to take full ownership of their data pipelines.
Our 100+ users, spread across 10+ teams, own in total more than 200 dags and 4000 tasks. We use a single Airflow instance with many cross-dag and cross-stream dependencies within these 200 dags. As it’s impossible to keep track of all 4000 tasks as a single entity, these pipelines can only be managed by the teams themselves.


## Data Science Platform at PlayStation and Apache Airflow
[URL](https://airflowsummit.org/sessions/2022/data-science-platform-at-playstation/)  
Date: 2022-05-26T21:30:00.000Z
Summary:
In this talk, we explain how Apache Airflow is at the center of our Kubernetes-based Data Science Platform at PlayStation. We talk about how we built a flexible development environment for Data Scientists to interact with Apache Airflow and explain the tools and processes we built to help Data Scientists promote their dags from development to production.
We will also talk about the impact of containerization and the usage of KubernetesOperator and the new SparkKubernetesOperator and the benefits of deploying Airflow in Kubernetes using the KubernetesExecutor across multiple environments.


## What's New with Amazon Managed Workflows for Apache Airflow (MWAA)
[URL](https://airflowsummit.org/sessions/2022/whats-new-with-mwaa/)  
Date: 2022-05-27T00:00:00.000Z
Summary:
In this session we will discuss the latest features of Amazon Managed Workflows for Apache Airflow (MWAA) as well as some tips and tricks to get the most out of the service. We’ll also discuss the AWS commitment to the Apache Airflow project and what we’re doing to stay connected and contribute to the community.


## Implementing Event-Based DAGs with Airflow
[URL](https://airflowsummit.org/sessions/2022/implementing-event-based-dags-with-airflow/)  
Date: 2022-05-27T01:00:00.000Z
Summary:
Needing to trigger DAGs based on external criteria is a common use case for data engineers, data scientists, and data analysts. Most Airflow users are probably aware of the concept of sensors and how they can be used to run your DAGs off of a standard schedule, but sensors are only one of multiple methods available to implement event-based DAGs. In this session, we’ll discuss different ways of implementing event-based DAGs using Airflow 2 features like the API and deferrable operators, with a focus on how to determine which method is the most efficient, scalable, and cost-friendly for your use case.


## Kyte: Scalable and Isolated DAG Development Experience at Lyft
[URL](https://airflowsummit.org/sessions/2022/kyte-dag-development-experience-at-lyft/)  
Date: 2022-05-27T01:30:00.000Z
Summary:
Developer velocity starts to become an issue as your user base grows and becomes more varied. This is compounded by the fact that it’s not easy to end-to-end test data pipelines as part of continuous integration. In this talk, we’ll go over what we’ve done at Lyft to make an effective development and testing environment, serving over 1000 users who have made over 5000 dags, at a rate of about 50 developer per week.


## Data Lineage with Apache Airflow and Apache Spark
[URL](https://airflowsummit.org/sessions/2022/data-lineage-with-apache-airflow-and-apache-spark/)  
Date: 2022-05-27T02:00:00.000Z
Summary:
Data within today’s organizations has become increasingly distributed and heterogeneous. It can’t be contained within a single brain, a single team, or a single platform…but it still needs to be comprehensible, especially when something unexpected happens. Data lineage can help by tracing the relationships between datasets and providing a cohesive graph that places them in context.
OpenLineage provides a standard for lineage collection that spans multiple platforms, including Apache Airflow and Apache Spark. In this session, Michael Collado from Datakin will show how to trace data lineage and useful operational metadata in Apache Spark and Airflow pipelines, and talk about how OpenLineage fits in the context of data pipeline operations and provides insight into the larger data ecosystem.


## What's new in Airflow 2.3?
[URL](https://airflowsummit.org/sessions/2022/whats-new-airflow-2-3/)  
Date: 2022-05-27T02:30:00.000Z
Summary:
This session will talk about the awesome new features the community has built that would be part of Airflow 2.3.
Highlights:
Dynamic Task Mapping DB. Downgrades Pruning old DB records Connections using JSON UI Improvements 

## Workshop: Contributing to Apache Airflow
[URL](https://airflowsummit.org/sessions/2022/workshop-contributing-to-apache-airflow/)  
Date: 2022-05-27T13:00:00.000Z
Summary:
Learn how to setup a development environment, how to pick your first issue, how to communicate effectively within the community and how to make your first PR.

## Workshop: Running Airflow within Cloud Composer
[URL](https://airflowsummit.org/sessions/2022/cloud-composer-workshop/)  
Date: 2022-05-27T14:30:00.000Z
Summary:
Hands on workshop showing how easy it is to deploy Airflow in a public Cloud. This workshop is mostly targeted at Airflow newbies and users who would like to learn more about Cloud Composer.

## Airflow and _____: A discussion around utilizing Airflow with other data tools
[URL](https://airflowsummit.org/sessions/2022/airflow-and___/)  
Date: 2022-05-27T15:00:00.000Z
Summary:
Come hang with Airflow practitioners from around the world using Airflow AND other data tools to power their data practice. From Databricks to Glue to Azure Data Factory, smart businesses make the right decision to standardize on Airflow for what it’s best at while using the other systems for what they are best at.


## Beyond Testing: How to Build Circuit Breakers with Airflow
[URL](https://airflowsummit.org/sessions/2022/beyond-testing-how-to-build-circuit-breakers-with-airflow/)  
Date: 2022-05-27T16:00:00.000Z
Summary:
Testing is an important part of the DataOps life cycle, giving teams confidence in the integrity of their data as it moves downstream to production systems. But what happens when testing doesn’t catch all of your bad data and “unknown unknown” data quality issues fall through the cracks? Fortunately, data engineers can apply a thing or two from DevOps best practices to tackle data quality at scale with circuit breakers, a novel approach to stopping bad data from actually entering your pipelines in the first place. In this talk, Prateek Chawla, Founding Team Member and Technical Lead at Monte Carlo, will discuss what circuit breakers are, how to integrate them with your Airflow DAGs, and what this looks like in practice. Time permitting, Prateek will also walk through how to build and automate Airflow circuit breakers across multiple cascading pipelines with Python and other common tools.


## TFX on Airflow with delegation of processing to third party services
[URL](https://airflowsummit.org/sessions/2022/tfx-on-airflow-with-delegationprocessing-to-third-party-services/)  
Date: 2022-05-27T16:00:00.000Z
Summary:
Learn how to externalize any TFX heavyweight computing outside Airflow, while maintaining Airflow as the orchestrator for your machine learning pipelines.

## Ingesting Game Telemetry in near Real-time dynamically into Redshift with Airflow (WB Games)
[URL](https://airflowsummit.org/sessions/2022/ingesting-game-telemetry/)  
Date: 2022-05-27T17:00:00.000Z
Summary:
We the Data Engineering Team here at WB Games implemented an internal Redshift Loader DAG(s) on Airflow that allow us to ingest data in near real-time at scale into Redshift, taking into account variable load on the DB and been able to quickly catch up data loads in case of various DB outages or high usage scenarios.
Highlights:
Handle any type of Redshift outages and system delays dynamically between multiple sources(S3) to sinks(Redshift). Auto tuning data copies for faster data backfill in case of delay without overwhelming commit queue. Supports schema evolution on Game data dynamically. Maintain data quality to ensure we do not create data gaps or dupes. Provide embedded custom metrics for deeper insights and anomaly detection. Airflow config based Declarative Dag implementation. 

## An Introduction to Data Lineage with Airflow and Marquez
[URL](https://airflowsummit.org/sessions/2022/an-introduction-to-data-lineage-with-airflow-and-marquez/)  
Date: 2022-05-27T17:30:00.000Z
Summary:
Learn how to collect and visualize lineage from a basic Airflow pipeline using Marquez. You will need to understand the basics of Airflow, but no experience with lineage is required.

## Hey maintainer (and user), exercise your empathy!
[URL](https://airflowsummit.org/sessions/2022/hey-maintainer-exercise-your-empathy/)  
Date: 2022-05-27T18:00:00.000Z
Summary:
This talk is a walk through throug a number of ways maintainers of open-source projects (for example Airflow) can improve the communication with their users by exercising empathy.
This subject is often overlooked in the cirriculum of average developer and contributor, but one that can make or break the product you developed, simply because it will become more approachable for users. Maintainers often forget or simply do not realize how many assumptions they have in their head.


## Talking With Management About Open Source
[URL](https://airflowsummit.org/sessions/2023/talking-management-open-source/)  
Date: 2023-09-19T09:00:00-04:00
Summary:
For those of us who already know how important open source is, it can be challenging to persuasively make the case to management, because we assume that everyone already knows the basics. This can work against us, confusing our audience and making us come across as condescending or concerned about irrelevant lofty philosophical points.
In this talk, we take it back to the basics. What does management actually need to know about open source, why it matters, and how to make decisions about consuming open source, contributing to open source, and open sourcing company code?


## Open Source is Pretty Secure, Actually
[URL](https://airflowsummit.org/sessions/2023/keynote-dustin-ingram/)  
Date: 2023-09-19T09:30:00-04:00
Summary:
We’ve heard a lot in the last few years about insecurity in the open source software ecosystem, whether it be vulnerabilities, supply chain attacks or malware. Has open source become suddenly fraught with security problems? Or is it maybe, possibly… actually doing great? Let’s delve into the collaborative nature of our open-source ecosystems, and explore how transparency, peer review, and community have created a robust security posture. We’ll examine real-world examples, dispel myths, and reveal the inherent strengths of open source in fostering a secure and resilient software ecosystem.


## The Faces of Airflow: A peek into the past and future
[URL](https://airflowsummit.org/sessions/2023/panels/panel-faces-airflow/)  
Date: 2023-09-19T10:00:00-05:00
Summary:
Airflow is almost 10 years old! Since starting out at AirBnB, the project has taken all sorts of twists and turns before getting to where it is now. Through its lifecycle, Airflow has seen an explosion of contributors (over 2400 and counting), end users, use cases, and so much more. This panel, moderated by Marc Lamberti, will be about some of the faces that have helped make Airflow what it is. You’ll hear from some of the most seasoned, and newest faces, about how the project has changed over the years, the community around it, and where they see it headed.


## Airflow at The Home Depot Canada: Observable orchestration platform for data integration and ML
[URL](https://airflowsummit.org/sessions/2023/airflow-at-the-home-depot-canada-observable-orchestration-platform-for-data-integration-and-ml/)  
Date: 2023-09-19T11:30:00-04:00
Summary:
The purpose of this session is to indicate how we leverage airflow in a federated way across all our business units to perform a cost-effective platform that accommodates different patterns of data integration, replication and ML tasks in a flexible way providing DevOps tunning of DAGs across environments that integrate to our open-source observability strategy that allows our SREs to have a consistent metrics, monitoring and alerting of data tasks.


## Airflow: Under the hood
[URL](https://airflowsummit.org/sessions/2023/airflow-under-the-hood/)  
Date: 2023-09-19T11:30:00-04:00
Summary:
Making a contribution to or becoming a committer on Airflow can be a daunting task, even for experienced Python developers and Airflow users. The sheer size and complexity of the code base may discourage potential contributors from taking the first steps. To help alleviate this issue, this session is designed to provide a better understanding of how Airflow works and build confidence in getting started.
During the session, we will introduce the main components of Airflow, including the Web Server, Scheduler, and Workers. We will also cover key concepts such as DAGs, DAG-run objects, Tasks, and Task Instances. Additionally, we will explain how tasks communicate with each other using XComs, and discuss the frequency of DAG runs based on the schedule. To showcase changes in the state of various objects, we will dive into the code level and continuously share the state of the database at every important checkpoint.


## Supporting the Vast Airflow Community: Lessons learned from over 100 Airflow webinars
[URL](https://airflowsummit.org/sessions/2023/supporting-the-vast-airflow-community-lessons-learned-from-over-100-airflow-webinars/)  
Date: 2023-09-19T11:30:00-04:00
Summary:
Astronomer has hosted over 100 Airflow webinars designed to educate and inform the community on best practices, use cases, and new features. The goal of these events is to increase Airflow’s adoption and ensure everybody, from new users to experienced power users, can keep up with a project that is evolving faster than ever. When new releases come out every few months, it can be easy to get stuck in past versions of Airflow. Instead, we want existing users to know how new features can make their lives easier, new users to know that Airflow can support their use case, and everybody to know how to implement the features they need and get them to production. This talk will cover some of the key learnings we’ve gathered from 2.5 years of conducting webinars aimed at supporting the community in growing their Airflow use, including how to best cater DevRel efforts to the many different types of Airflow users and how to effectively push for the adoption of new Airflow features.


## What Everybody Ought to Know About Airflow
[URL](https://airflowsummit.org/sessions/2023/what-everybody-ought-to-know-about-airflow/)  
Date: 2023-09-19T11:30:00-04:00
Summary:
Airflow is a powerful tool for orchestrating complex data workflows, which have undergone significant changes over the past two years.
Since the Airflow release cycle has accelerated, you may struggle to keep up with the continuous flow of new features and improvements, which can lead to miss opportunities for addressing new use cases or solving your existing ones more efficiently.
This presentation is intended to give you a solid update on the possibilities of Airflow and address misconceptions you may have heard or still believe that used to be valid but no longer are.


## Airflow at Asurion: Simplified orchestration at petabyte scale
[URL](https://airflowsummit.org/sessions/2023/airflow-at-asurion-simplified-orchestration-at-petabyte-scale/)  
Date: 2023-09-19T12:00:00-04:00
Summary:
Workload Orchestration is at the heart of a successful Data lakehouse implementation. Especially for the “house” part which represents the Datawarehouse workloads which often are complex because of the very nature of warehouse data, which have dependency orchestration problems.
We at Asurion have spent years in perfecting the Airflow solution to make it a super power for our Data Engineers. We have innovated in key areas like single operator for all use cases, auto DAG code generation, custom UI components for Data Engineers, monitoring tools etc. With over a few million job runs per year running on a platform with over 3 nines of availability, we have condensed years of our learnings into valuable ideas that can inspire and help all other Data enthusiasts.


## Airflow Executors: Past, present and future
[URL](https://airflowsummit.org/sessions/2023/airflow-executors-past-present-and-future/)  
Date: 2023-09-19T12:00:00-04:00
Summary:
Executors are a core concept in Apache Airflow and are an essential piece to the execution of DAGs. They have seen a lot of investment over the year and there are many exciting advancements that will benefit both users and contributors.
This talk will briefly discuss executors, how they work and what they are responsible for. It will then describe Executor Decoupling (AIP-51) and how this has fully unlocked development of third-party executors. We’ll touch on the migration of “core” executors (such as Celery and Kubernetes) to their own package as well as the addition of new “3rd party” executors from providers such as AWS. Finally, a description/demo of Hybrid Executors, a proposed new feature to allow multiple executors to be used natively and seamlessly side by side within a single Airflow environment; which will be a powerful feature in a future full of many new Airflow executors.


## Introducing airflowctl: A CLI to streamline getting started with Airflow
[URL](https://airflowsummit.org/sessions/2023/introducing-airflowctl/)  
Date: 2023-09-19T12:00:00-04:00
Summary:
New users starting with Airflow frequently encounter several challenges, ranging from the complexities of Containers and virtual environments to the Python dependency hell. Moreover, their familiarity with tools such as Docker, docker-compose, and Helm might be somewhat limited and even overkill. In contrast, seasoned Airflow users encounter their problems, encompassing configuration conflics with ongoing Airflow projects and intricacies stemming from Docker and docker-compose configurations and lack of visibility into all the projects.


## Opportunities to join the Airflow (docs) community
[URL](https://airflowsummit.org/sessions/2023/opportunities-to-join-the-airflow-docs-community/)  
Date: 2023-09-19T12:00:00-04:00
Summary:
Open Source doc edits provide a low-stakes way for new users to first contribute. Ideally, new users find opportunities and feel welcome to fix docs as they learn, engaging with the community from the start. But, I found that contributing docs to Airflow had some surprising obstacles.
In this talk, I’ll share my first docs contribution journey, including problems and fixes. For example, you must understand how Airflow uses Sphinx and know when to choose to edit in the GitHub UI or locally. But it wasn’t documented that GitHub renders only Markdown previews and since Sphinx uses markup, you must build docs locally to check formatting; an opportunity for me to add to the Contributor Guide for docs.


## Demystifying Apache Airflow: Separating facts from fiction
[URL](https://airflowsummit.org/sessions/2023/demystifying-apache-airflow-separating-facts-from-fiction/)  
Date: 2023-09-19T12:30:00-04:00
Summary:
Apache Airflow is a popular workflow platform, but it often faces critiques that may not paint the whole picture. In this talk, we will unpack the critiques of Apache Airflow and provide a balanced analysis. We will highlight the areas where these critiques correctly point out Airflow’s weaknesses, debunk common myths, and showcase where competitors like Dagster and Prefect are excelling.
By understanding the pros and cons of Apache Airflow, attendees will be better equipped to make informed decisions about whether Airflow is the right choice for their use cases. This talk will provide a comprehensive and objective assessment of Apache Airflow and its place in the workflow management ecosystem.


## Empowering Collaborative Data Workflows with Airflow and Cloud Services
[URL](https://airflowsummit.org/sessions/2023/empowering-collaborative-data-workflows-with-airflow-and-cloud-services/)  
Date: 2023-09-19T12:30:00-04:00
Summary:
Productive cross-team collaboration between data engineers and analysts is the goal of all data teams, however, fulfilling on that mission can be challenging given the diverse set of skills that each group brings. In this talk we present an example of how one team tackled this topic by creating a flexible, dynamic and extensible framework using Airflow and cloud services that allowed engineers and analysts to jointly create data-centric micro-services to serve up projections and other robust analysis for use in the organization. The framework, which utilized dynamic DAG generation configured using yaml files, Kubernetes jobs and dbt transformations, abstracted away many of the details associated with workflow orchestration, allowing analysts to focus on their Python or R code and data processing logic while enabling data engineers to monitor the pipelines and ensure their scalability.


## Future of the Airflow UI
[URL](https://airflowsummit.org/sessions/2023/future-of-the-airflow-ui/)  
Date: 2023-09-19T12:30:00-04:00
Summary:
We are continuing to modernize the Airflow UI to make it easier to manage all aspects of your DAGs. See a demo of the latest updates and improve your workflows with new tips and tricks. Then get a preview of what else will be coming soon.
Followed up by Q&A for people to field their own use-cases and explore new ideas on how to improve the user experience.


## Airflow at Reddit: How we migrated from Airflow 1 to Airflow 2
[URL](https://airflowsummit.org/sessions/2023/airflow-at-reddit-how-we-migrated-from-airflow-1-to-airflow-2/)  
Date: 2023-09-19T14:00:00-04:00
Summary:
We would love to speak about our experience upgrading our old airflow 1 infrastructure to airflow 2 on kubernetes and how we orchestrated the migration of approximately 1500 DAGs that were owned by multiple teams in our organization. We had some interesting challenges along the way and can speak about our solutions.
Points we can talk about:
Old airflow 1 infrastructure and why we decided to move to kubernetes for airflow 2. Possible migration paths we thought of and why we chose the route we did. Things we did to make the migration easier to achieve:


## Building a Commercial Service with Open-Source Community Focus, presented by AWS
[URL](https://airflowsummit.org/sessions/2023/building-a-commercial-service-with-open-source-community-focus/)  
Date: 2023-09-19T14:00:00-04:00
Summary:
Amazon Managed Workflows for Apache Airflow (MWAA) was released in November 2020. Throughout MWAA’s design we held the tenets that this service would be open-source first, not forking or deviating from the project, and that the MWAA team would focus on improving Airflow for everyone—whether they run Airflow on MWAA, on AWS, or anywhere else.
This talk will cover some of the design choices made to facilitate those tenets, how the organization was set up to contribute back to the community, what those contributions look like today, how we’re getting those contributions in the hands of users, and our vision for future engagement with the community.


## Guided Tour to DAG Authoring
[URL](https://airflowsummit.org/sessions/2023/guided-tour-to-dag-authoring/)  
Date: 2023-09-19T14:00:00-04:00
Summary:
New to Airflow or haven’t followed any of the recent DAG authoring enhancements? This talk is for you!
We will go through various DAG authoring features like Setup/Teardown tasks (~2.7), Datasets (2.4), Dynamic Tasks (2.3) and Async tasks (2.2). You won’t be an expert after this short talk, however, you’ll have a head start when you write your next DAG, no hacks required.


## Operators Need to Die
[URL](https://airflowsummit.org/sessions/2023/operators-need-to-die/)  
Date: 2023-09-19T14:00:00-04:00
Summary:
Operators form the core of the language of Airflow. In this talk I will argue that while they have served their purpose, they are holding back the development of Airflow and if Airflow wants to stay relevant in the world of the ’new’ data stack (hint: it isn’t currently considered to be part of it) self-service data mesh it needs to kill its darling.


## Airflow at Bloomberg: Leveraging dynamic DAGs for data ingestion
[URL](https://airflowsummit.org/sessions/2023/airflow-at-bloomberg-leveraging-dynamic-dags-for-data-ingestion/)  
Date: 2023-09-19T14:30:00-04:00
Summary:
Bloomberg’s Data Platform Engineering team powers some of the most valuable business and financial data on which Bloomberg clients rely. We recently built a configuration-driven system that allows non-engineers to onboard alternative datasets into the company’s ecosystem. This system uses Apache Airflow to orchestrate the data flow across different applications and Bloomberg Terminal functions. We are unique in that we have over 1500 dynamic DAGs tailored for each dataset’s needs (which very few Airflow users have). In this talk, we will review our high-level Airflow architecture, how we leverage the dynamic DAGs in our ETL pipeline, as well as review some of the challenges we faced.


## DAG Authoring without PhD, presented by Google Cloud
[URL](https://airflowsummit.org/sessions/2023/dag-authoring-without-phd/)  
Date: 2023-09-19T14:30:00-04:00
Summary:
DAG Authoring - learn how to go beyond the basics and best practices when implementing Airflow DAGs. It will be a survival guide for Airflow DAG developers who need to cope with hundreds of Airflow operators.
This session will go beyond 101 or “for dummies” session and will be of interest to both those who are just starting to develop Airflow DAGs and Airflow experts, as it will help them improve their productivity.


## Event-based DAG Parsing: No more F5ing in the UI
[URL](https://airflowsummit.org/sessions/2023/event-based-dag-parsing-no-more-f5ing-in-the-ui/)  
Date: 2023-09-19T14:30:00-04:00
Summary:
Have you ever added a DAG file and had no clue what happened to it? You’re not alone! With default settings, Airflow can wait up to 5 minutes before processing new DAG files.
In this talk, I’ll discuss the implementation of an event-based DAG parser that immediately processes changes in the DAGs folder. As a result, changes are reflected immediately in the Airflow UI. In this talk I will cover:


## My Journey to Committer Status: What I learned and how it can help you
[URL](https://airflowsummit.org/sessions/2023/my-journey-to-committer-status-what-i-learned-and-how-it-can-help-you/)  
Date: 2023-09-19T14:30:00-04:00
Summary:
Apache Airflow is one of the largest Apache projects by many metrics but it ranks particularly high in the number of contributors involved in the project. This leads to hundreds of Github Issues, Pull Requests and Discussions being submitted to the project every month. So it is critical to have an ample number of Committers to support the community.
In this talk I will summarize my personal experience working towards, and ultimately achieving, committer status in Apache Airflow. I’ll cover the lessons I learned along the way as well as provide some advice and best practices to help others achieve committer status themselves.


## Apache Airflow and OpenTelemetry
[URL](https://airflowsummit.org/sessions/2023/apache-airflow-and-opentelemetry/)  
Date: 2023-09-19T15:00:00-04:00
Summary:
OpenTelemetry is a vendor-neutral open-source (CNCF) observability framework that is supported by many vendors industry-wide. It is used for instrumenting, generation, collection, and exporting of data within systems which then are ingested by analytics tools that can provide tracing, metrics, and logs. It has long been the plan to adopt the OTel standard within Airflow, allowing builders and users to take advantage of valuable data that could help improve the efficiency, cost and performance of their systems. Let us talk about the journey that started a few years ago to bring this dream to reality.


## Forging the Future: Five years of fabricating with Airflow
[URL](https://airflowsummit.org/sessions/2023/forging-the-future-five-years-of-fabricating-with-airflow/)  
Date: 2023-09-19T15:00:00-04:00
Summary:
As a data engineer, I’ve used Airflow extensively over the last 5 years: across 3 jobs, several different roles; for side projects, for critical infrastructure; for manually triggered jobs, for automated workflows; for IT (Ookla/Speedtest.net), for science (Allen Institute for Cell Science), for the commons (Openverse), for liberation (Orca Collective). Authoring a DAG has changed dramatically since 2018, thanks to improvements in both Airflow and the Python language. In this session, we’ll take a trip back in time to see how DAGs looked several years ago, and what the same DAGs might look like now. We’ll appreciate the many improvements that have been made towards simplifying workflow construction. I’ll also discuss the significant advancements that have been made around deploying Airflow. Lastly, I’ll give a brief overview of different use cases and ways I’ve seen Airflow leveraged.


## Reliable Airflow DAG Design When Building a Time-Series Data Lakehouse
[URL](https://airflowsummit.org/sessions/2023/reliable-airflow-dag-design-when-building-a-time-series-data-lakehouse/)  
Date: 2023-09-19T15:00:00-04:00
Summary:
As a team that has built a Time-Series Data Lakehouse at Bloomberg, we looked for a workflow orchestration tool that could address our growing scheduling requirements. We needed a tool that was reliable and scalable, but also could alert on failures and delays to enable users to recover quickly from them. From using triggers over simple sensors to implementing custom SLA monitoring operators, we explore our choices in designing Airflow DAGs to create a reliable data delivery pipeline that is optimized for failure detection and remediation.


## The Why and How of Running a Self-Managed Airflow on Kubernetes
[URL](https://airflowsummit.org/sessions/2023/the-why-and-how-of-running-a-self-managed-airflow-on-kubernetes/)  
Date: 2023-09-19T15:00:00-04:00
Summary:
Today, all major cloud service providers and 3rd party providers include Apache Airflow as a managed service offering in their portfolios. While these cloud based solutions help with the undifferentiated heavy lifting of environment management, some data teams are also looking to operate self-managed Airflow instances to satisfy specific differentiated capabilities. In this session, we would talk about:
Why should you might need to run self managed Airflow
The available deployment options (with emphasis on Airflow on Kubernetes)


## Airflow at Salesforce: Building a fully managed workflow orchestration system
[URL](https://airflowsummit.org/sessions/2023/airflow-at-salesforce-building-a-fully-managed-workflow-orchestration-system/)  
Date: 2023-09-19T15:30:00-04:00
Summary:
In this presentation, we discuss how we built a fully managed workflow orchestration system at Salesforce using Apache Airflow to facilitate dependable data lake infrastructure on the public cloud. We touch upon how we utilized kubernetes for increased scalability and resilience, as well as the most effective approaches for managing and scaling data pipelines. We will also talk about how we addressed data security and privacy, multitenancy, and interoperability with other internal systems.


## Flexible DAG Trigger Forms (AIP-50)
[URL](https://airflowsummit.org/sessions/2023/flexible-dag-trigger-forms-aip-50/)  
Date: 2023-09-19T15:30:00-04:00
Summary:
As user of Airflow we often use DagRun.conf attributes to control content and flow of a DAG run. Previously the Airflow UI only allowed to launch via JSON in the UI. This was technically feasible but not user friendly. A user needs to model, check and understand the JSON and enter parameters manually without the option to validate before trigger.
Similar like Jenkins or Github/Azure pipelines we desire an UI option to trigger with a UI and specifying parameters.


## Mastering Dependencies: The Airflow way
[URL](https://airflowsummit.org/sessions/2023/mastering-dependencies-the-airflow-way/)  
Date: 2023-09-19T15:30:00-04:00
Summary:
Apache Airflow has over 650 Python dependencies. In case you did not know already, dependencies in Python are difficult subject. And Airflow has its own, custom ways of managing the dependencies.
Airflow has a rather complex system to manage dependencies in their CI system, but this talk is not about it. This talk is directed to the users of Airflow who want to keep their dependencies updated, describing ways they can do it.


## Things to Consider When Building an Airflow Service
[URL](https://airflowsummit.org/sessions/2023/things-to-consider-when-building-an-airflow-service/)  
Date: 2023-09-19T15:30:00-04:00
Summary:
Data platform teams often find themselves in a situation where they have to provide Airflow as a service to downstream teams, as more users and use cases in their organization require an orchestrator. In these situations, it’s giving each team it’s own Airflow environment can unlock velocity and actually be lower overhead to maintain than a monolithic environment.
This talk will be about things to keep in mind when building an Airflow service that supports several environments, persona of users, and use cases. Namely, we’ll discuss principles to keep in mind when balancing centralized control over the data platform with decentralized teams using Airflow in a way that they’ll need. This will include things around observability, developer productivity, security, and infrastructure. We’ll also talk about day 2 concerns around overheard, infrastructure maintenance, and other tradeoffs to consider.


## A Hypervisor for Airflow, presented by Astronomer
[URL](https://airflowsummit.org/sessions/2023/airflow-everyone/)  
Date: 2023-09-19T16:15:00-04:00
Summary:
Over the last few years, we’ve spent countless hours talking to data engineers from everywhere from Fortune 500s to seed stage startups. In doing so, we’ve learned all about what it takes to deliver a world class Airflow service perfect for everyone. We’ve packaged all that up into The Astro Hypervisor, a new part of our platform that gives users a whole new level of control in Airflow.
We’ll talk through how we’ve built this hypervisor and how our customers will be able to use it for autoscaling, tracking the health of Airflow environments and so much more.


## Better Support for Using Multiple Namespaces with KubernetesExecutor
[URL](https://airflowsummit.org/sessions/2023/better-support-for-using-multiple-namespaces-with-kubernetesexecutor/)  
Date: 2023-09-19T16:15:00-04:00
Summary:
Airflow’s KubernetesExecutor has supported multi_namespace_mode for long time. This feature is great at allowing Airflow jobs to run in different namespaces on the same Kubernetes clusters for better isolation and easier management.
However, this feature requires cluster-role for the Airflow scheduler, which can create security problems or be a blocker for some users.
PR https://github.com/apache/airflow/pull/28047 , which will become available in Airflow 2.6.0, resolves this issue by allowing Airflow users to specify multi_namespace_mode_namespace_list when using multi_namespace_mode, so that no cluster-role is needed and user only needs to ensure the Scheduler has permissions to certain namespaces rather than all namespaces on the Kubernetes cluster.


## From Bug To Bug Fix: Tips of how to submit an issue
[URL](https://airflowsummit.org/sessions/2023/from-bug-to-bug-fix-tips-of-how-to-submit-an-issue/)  
Date: 2023-09-19T16:15:00-04:00
Summary:
How to submit an Issue for community to fix
To ensure a quality product the Airflow community relies on bug reports from Airflow users. Often times bug reports are incomplete or fail to include steps for observed bug to be re-created.
This workshop will present an example of bug-to-issue process, namely, how to rule out non-Airflow issues and once an Airflow issue is suspected, how to submit an issue for community to see. This could also cover how the community picks up on an issue and eventually fixes it in a future release.


## Migrating from Enterprise Scheduler like Autosys, TIDAL, Stonebranch to Airflow
[URL](https://airflowsummit.org/sessions/2023/migrating-from-enterprise-scheduler-like-autosys-tidal-stonebranch-to-airflow/)  
Date: 2023-09-19T16:15:00-04:00
Summary:
How we migrated from Autosys with 1000s of jobs with 800+ dependencies with SLA to be met every hour in a Canada Prominent Bank.
Use case to migrate from enterprise scheduler
$ spent for every license and renewal cost
SLA,Monitoring,Auditing,Devops Integration
Vendor lockin
4.Integration to multiple providers


## Airflow at Snap: Managing permissions, migrations and internal tools
[URL](https://airflowsummit.org/sessions/2023/airflow-at-snap-managing-permissions-migrations-and-internal-tools/)  
Date: 2023-09-19T16:45:00-04:00
Summary:
We will cover how Snap (parent company of Snapchat) has been using Airflow since 2016.
How we built a secure deployment on GCP that integrates with internal tools for workload authorization, RBAC and more. We made permissions for DAGs easy to use for customers using k8s workload identity binding and tight UI integration.
How are we migrating 2500+ DAGs from Airflow V1, Python 2 to V2 Python 3 using tools + automations. Making code/DAG migration requires significant amount of time investment. Our team created several tools that can convert or re-write DAGs in the new format.


## An Introduction to Airflow Cluster Policies
[URL](https://airflowsummit.org/sessions/2023/an-introduction-to-airflow-cluster-policies/)  
Date: 2023-09-19T16:45:00-04:00
Summary:
Cluster Policies are an advanced Airflow feature composed of a set of hooks that allow cluster administrators to implement checks and mutations against certain core Airflow constructs (DAGs, Tasks, Task Instances, Pods).
In this talk, we will discuss how cluster administrators can leverage these functions in order to better govern the workloads that are running in their environments.


## How to Build a System Test Dashboard and why you Should do it?
[URL](https://airflowsummit.org/sessions/2023/how-to-build-a-system-test-dashboard-and-why-you-should-do-it/)  
Date: 2023-09-19T16:45:00-04:00
Summary:
System tests are executable DAGs for example and testing purposes. With a simple pytest command, you can run an entire DAG. From a provider point of view, they can be viewed as integration tests for all provider related operators and sensors. Running these system tests frequently and monitoring the results allow us to enforce stability amongst many other benefits. In this presentation we will explore how AWS built their system test environment, from the GitHub fork to the health dashboard that exists today…but more importantly, why you should do it as well!


## To Debug a DAG: The Airflow local dev story
[URL](https://airflowsummit.org/sessions/2023/to-debug-a-dag-the-airflow-local-dev-story/)  
Date: 2023-09-19T16:45:00-04:00
Summary:
As much as we love airflow, local development has been a bit of a white whale through much of its history. Until recently, Airflow’s local development experience has been hindered by the need to spin up a scheduler and webserver. In this talk, we will explore the latest innovation in Airflow local development, namely the “dag.test()” functionality introduced in Airflow 2.5.
We will delve into practical applications of “dag.test()”, which empowers users to locally run and debug Airflow DAGs on a single python process. This new functionality significantly improves the development experience, enabling faster iteration and deployment.


## Airflow at Delivery Hero: Running a data mesh with ~500 Airflow instances
[URL](https://airflowsummit.org/sessions/2023/airflow-at-delivery-hero-running-a-data-mesh-with-500-airflow-instances/)  
Date: 2023-09-19T17:15:00-04:00
Summary:
Ever thought how airflow could play a pivotal role in data mesh architecture, hosting thousands of DAGs and hundreds of thousands daily running tasks, let’s find out!
Delivery Hero delivers food in 70 countries with 12 different brands and platforms. With thousands of engineers, analysts and data scientists spread across many countries running analytics and ML services for all these orders delivered. Serving the workflow orchestration needs for such a massive group becomes a challenge.


## DAG Parsing Optimizations
[URL](https://airflowsummit.org/sessions/2023/dag-parsing-optimizations/)  
Date: 2023-09-19T17:15:00-04:00
Summary:
As big Airflow users grow their usage to hundreds of DAGs, parsing them can become a performance bottleneck in the scheduler.
In this talk, we’ll explore how this situation was improved by using caching techniques and pre-processing of DAGs to minimize the overhead of parsing them at runtime.
We’ll also touch on how the the performance of the existing code was analyzed to find points of improvement.
We may include a section on how to configure airflow to benefit from those recent changes, and some tips on how to make DAGs that are quick to parse, but this will not be the core of the talk.


## Nurturing an Open Source Community is Like Tending a Garden
[URL](https://airflowsummit.org/sessions/2023/nurturing-open-source-community/)  
Date: 2023-09-19T17:15:00-04:00
Summary:
Nurturing a healthy open source community is hard work and requires discipline. There’s a lot of inherent friction in a distributed community that makes it difficult for the many participants to collaborate. It is challenging to contribute something when you’re new to a community. It is overwhelming for maintainers to give enough attention to contributors when a project becomes popular. In this talk, I’ll go over the common pitfalls of open source communities and will go over practices that contribute to keeping your community healthy.


## Testing Airflow DAGs with Dagtest
[URL](https://airflowsummit.org/sessions/2023/testing-airflow-dags-with-dagtest/)  
Date: 2023-09-19T17:15:00-04:00
Summary:
For the dag owner, testing Airflow DAGs can be complicated and tedious. Kubectl cp your dag from local to pod, exec into the pod, and run a command? Install breeze? Why pull the Airflow image and start up the webserver / scheduler / triggerer if all we want is to test the addition of a new task?
It doesn’t have to be this hard. At Etsy, we’ve simplified testing dags for the dag owner with dagtest. Dagtest is a Python package that we house on our internal PyPi. It is a small client binary that makes HTTP requests to a test API. The test API is a simple Flask server that receives these requests and builds pods to run airflow dags backfill commands based on the options provided via dagtest. The simplest of these is a dry-run. Typically, users run test runs where the dag executes E2E for a single ds.


## Airflow at StyleSeat: Our journey, challenges & results
[URL](https://airflowsummit.org/sessions/2023/airflow-at-styleseat-our-journey-challenges-results/)  
Date: 2023-09-19T17:45:00-04:00
Summary:
We will share the case study of Airflow at StyleSeat, where within a year our data grew from 2 million data points per day to 200 million. Our original solution for orchestrating this data was not enough, so we migrated to an Airflow based solution.
Previous implementation Our tasks were orchestrated with hourly triggers on AWS Cloudwatch rules in their own log groups. Each task was a lambda individually defined as a task and executed python code from a docker image. As complexity increased, there were frequent downtimes and manual executions for failed tasks and their downstream dependencies.With every downtime, our business stakeholders started losing trust in Data and recovery times were longer with each downtime.


## Building an Open Source Data Warehouse
[URL](https://airflowsummit.org/sessions/2023/building-an-open-source-data-warehouse/)  
Date: 2023-09-19T17:45:00-04:00
Summary:
Volunteers in Saint Louis are using Airflow to build an open source data warehouse of real estate data (permits, assessments, violations, etc), with an eye towards creating a national open data standard. This talk will focus on the unique challenges of running an open source data warehouse, and what it looks like to work with volunteers to create data pipelines.


## Multi-tenancy State of the Union
[URL](https://airflowsummit.org/sessions/2023/multi-tenancy-state-of-the-union/)  
Date: 2023-09-19T17:45:00-04:00
Summary:
This sesion is about the current state of implementation for multi-tenancy feature of Airflow. This is a long-term feature that involves multiple changes, separate AIPs to implement, with the long-term vision of having single Airflow instance supporting multiple, independed teams using it - either from the same company or as part of Airflow-As-A-Service implementation.


## Reducing Cost with Async/Deferrable Operators
[URL](https://airflowsummit.org/sessions/2023/reducing-cost-with-async-deferrable-operators/)  
Date: 2023-09-19T17:45:00-04:00
Summary:
At Condé Nast, we have heavily leveraged async/deferrable operators to reduce our Airflow-associated costs. By implementing async/deferrable operators in all of our pipelines, we have been able to realize a cost reduction of 54% compared with our previous usage of non-async/deferrable operators.


## Building and deploying LLM applications with Apache Airflow
[URL](https://airflowsummit.org/sessions/2023/keynote-llm/)  
Date: 2023-09-20T09:00:00-04:00
Summary:
Behind the growing interest in Generate AI and LLM-based enterprise applications lies an expanded set of requirements for data integrations and ML orchestration. Enterprises want to use proprietary data to power LLM-based applications that create new business value, but they face challenges in moving beyond experimentation. The pipelines that power these models need to run reliably at scale, bringing together data from many sources and reacting continuously to changing conditions.


## AI/ML is Changing Orchestration: How Kubernetes can accelerate Airflow
[URL](https://airflowsummit.org/sessions/2023/keynote-clayton-coleman/)  
Date: 2023-09-20T09:30:00-04:00
Summary:
It should be no surprise to the Airflow community that the hype around generative large language models (LLMs) and their wildly-inventive chat front ends have brought significant attention to growing these models and feeding them on a steady diet of data. For many communities in the infrastructure, orchestration, and data landscape this is an opportunity to think big, help our users scale, and make the right foundational investments to sustain that growth over the long term.


## The Future of Airflow: What Users Want
[URL](https://airflowsummit.org/sessions/2023/panels/panel-users-want/)  
Date: 2023-09-20T10:00:00-05:00
Summary:
There are so many ways to run Airflow and as such, lots of folks responsible for running Airflow for downstream users. While many orgs go with a managed service(Astronomer, AWS MWAA, Google Cloud Composer), many also prefer running an Airflow platform themselves. This panel will be about what users want from a managed Airflow service, from the perspective of those charged with providing one. We’ll talk about use cases, roadmaps, and best practices that’ve been accumulated along the way.


## A Single Pane of Glass on Airflow using Astro Python SDK, Snowflake, dbt, and Cosmos
[URL](https://airflowsummit.org/sessions/2023/a-single-pane-of-glass-on-airflow-using-astro-python-sdk-snowflake-dbt-and-cosmos/)  
Date: 2023-09-20T11:30:00-04:00
Summary:
ETL data pipelines are the bread and butter of data teams that must design, develop, and author DAGs to accommodate the various business requirements.
dbt is becoming one of the most used tools to perform SQL transformations on the Data Warehouse, allowing teams to harness the power of queries at scale.
Airflow users are constantly finding new ways to integrate dbt with the Airflow ecosystem and build a single pane of glass where Data Engineers can manage and administer their pipelines.


## Cross Environment Event Based Triggers with Airflow
[URL](https://airflowsummit.org/sessions/2023/cross-environment-event-based-triggers-with-airflow/)  
Date: 2023-09-20T11:30:00-04:00
Summary:
In an environment with multiple Airflow instances , how we build custom operators and framework to share events across the instances and trigger dags based on those events


## Data about data: the case for a privacy centric schema language
[URL](https://airflowsummit.org/sessions/2023/data-about-data/)  
Date: 2023-09-20T11:30:00-04:00
Summary:
This talk is speculative: orchestration tools like Airflow have it made it very easy to pull and push data from anywhere to everywhere. But we don’t know what data we are pushing around.
What if we have a schema language that we could use to describe this data? Not in terms of data type but in terms of sensitivity and instructions on how to handle this?
This talk is about the headaches companies are facing day to day and that maybe there’s an opportunity for the Airflow community to help solve this problem.


## Delay Modeling and DAG Connectivity: Optimizing Airflow performance in large organizations
[URL](https://airflowsummit.org/sessions/2023/delay-modeling-and-dag-connectivity-optimizing-airflow-performance-in-large-organizations/)  
Date: 2023-09-20T11:30:00-04:00
Summary:
In large organizations, data workflows can be complex and interconnected, with multiple dependencies and varied runtime requirements. To ensure efficient and timely execution of workflows, it is important to understand the factors that affect the performance of the system, such as network congestion, resource availability, and DAG structure. In this talk, we will explore how delay modeling and DAG connectivity analysis can be used to optimize Airflow performance in large organizations. We will present a network analysis of an airflow instance with multiple interconnected DAGs, and demonstrate how delay modeling can be used to estimate maximum delay and identify bottlenecks in the system. We will also discuss how the delay model can be used to optimize runtime and improve overall system performance.


## Airflow at Faire: Democratizing ML feature store framework at scale
[URL](https://airflowsummit.org/sessions/2023/airflow-at-faire-democratizing-ml-feature-store-framework-at-scale/)  
Date: 2023-09-20T12:00:00-04:00
Summary:
Data science and machine learning are at the heart of Faire’s industry-celebrated marketplace (a16z top-ranked marketplace) and drive powerful search, navigation, and risk functions which are powered by ML models that are trained on 3000+ features defined by our data scientists.
Previously, defining, backfilling and maintaining feature lifecycle was error-prone. Having a framework built on top of Airflow has empowered them to maintain and deploy their changes independently.
We will explore:


## Better Airflow with Metaflow : A modern human-centric ML infrastructure stack
[URL](https://airflowsummit.org/sessions/2023/better-airflow-with-metaflow-a-modern-human-centric-ml-infrastructure-stack/)  
Date: 2023-09-20T12:00:00-04:00
Summary:
Airflow is a household brand in data engineering: It is readily familiar to most data engineers, quick to set up, and, as proven by millions of data pipelines powered by it since 2014, it can keep DAGs running.
But with the increasing demands of ML, there is a pressing need for tools that meet data scientists where they are and address two pressing issues - improving the developer experience & minimizing operational overhead.


## Circumventing Airflow's Limitations around Multitenancy
[URL](https://airflowsummit.org/sessions/2023/circumventing-airflows-limitations-around-multitenancy/)  
Date: 2023-09-20T12:00:00-04:00
Summary:
A steady rise in users and business critical workflows poses challenges to development and production workflows. The solution: enable multi-tenancy on our single Airflow instance. We needed to enable teams to manage their python requirements, and ensure DAGs were insulated from each other. To achieve this we divided our monolithic setup into three parts: Infrastructure (with common code packaging), Workspace Creation, and CI/CD to manage deployments.
Backstage templates enable teams to create isolated development environments that resemble our production environment, ensuring consistency. Distributing common code via a private pypi gives teams more control over what code their DAGs run. And a PythonOperator Shim in production utilizes virtualenv to run Python code with each team’s defined requirements for their DAG. In doing these things we enable effective multi-tenancy, and facilitate easier development and production workflows for Airflow.


## How to use Data Contracts for Data Quality in your Airflow Ecosystem
[URL](https://airflowsummit.org/sessions/2023/how-to-use-data-contracts-for-data-quality-in-your-airflow-ecosystem/)  
Date: 2023-09-20T12:00:00-04:00
Summary:
Data contracts have been much discussed in the community of late, with a lot of curiosity around how to approach this concept in practice.
We believe data contracts need a harmonizing layer to manage data quality in a uniform manner across a fragmented stack. We are calling this harmonizing layer the Control Plane for Data - powered by the common thread across these systems: metadata.
For teams already orchestrating pipelines with Airflow, data contacts can be an effective way to process data that meets preset quality standards. With a control plane as a connecting layer, producers can build data contracts that consumers can rely on, ensuring DAGs only run when a contract is valid. Producers can govern how workflows should behave, and consumers receive the tooling they need to only opt into high quality data.


## Airflow at GoDaddy: From on-prem to cloud to PaaS
[URL](https://airflowsummit.org/sessions/2023/airflow-at-godaddy-from-on-prem-to-cloud-to-paas/)  
Date: 2023-09-20T12:30:00-04:00
Summary:
Discover the transformation of Airflow at GoDaddy: from its initial deployment on-prem to its migration to the cloud, and finally to a Single Pane Orchestration Model.
This evolution has streamlined our Data Platform and improved governance. Our experience will be beneficial for anyone seeking to optimize their Airflow implementation and simplify their orchestration processes.
History and Use-cases Design, Organization decisions, and Governance: Examining the decision-making process and governance structure. Migration to Cloud:Process of transitioning Airflow from on-premises to the cloud. Data Processing engines used with Airflow for Data Processing. Challenges: Obstacles faced during and after migration and how they were overcome. *Demonstrating how Airflow can be integrated with a central Glue Catalog and Data Lake Mesh model. Single Pane Orchestration (PAAS) and custom re-usable Github Actions: Examining benefits of using a Single Pane Orchestration model Monitoring 

## Airflow Driven Data Lineage In Public Cloud
[URL](https://airflowsummit.org/sessions/2023/airflow-driven-data-lineage-in-public-cloud/)  
Date: 2023-09-20T12:30:00-04:00
Summary:
The session will cover capabilities of data lineage in Apache Airflow, how to use them, and motivations for it. It will present the technical know-how of integrating data lineage solutions with Apache Airflow, and provisioning DAGs metadata to fuel lineage functionalities in a way transparent to the user, limiting the setup friction. It will include Google’s Cloud Composer lineage integration implemented through the current Airflow’s data lineage architecture, and our approach to the lineage evolution strategy.


## Change Management Done Right Across Environments and Tools: DAGs, datasets and visualizations
[URL](https://airflowsummit.org/sessions/2023/change-management-done-right-across-environments-and-tools-dags-datasets-and-visualizations/)  
Date: 2023-09-20T12:30:00-04:00
Summary:
Change management in data teams can be challenging to say the least. Not only you have to evolve your data pipelines, data structures, and datasets themselves across environments, you also have to keep data exploration and visualizations tools in sync.
In this talk, we’ll be exploring how to do this best across environments (ie: dev, staging and prod), talking about how CI/CD can help, implementing good data ops practices and cranking up the level of rigor where it matters.


## Enabling Data Mesh by Moving from a Monolithic Airflow to Several Smaller Environments
[URL](https://airflowsummit.org/sessions/2023/enabling-data-mesh-by-moving-from-a-monolithic-airflow-to-several-smaller-environments/)  
Date: 2023-09-20T12:30:00-04:00
Summary:
Kiwi.com started using Airflow in June 2016 as an orchestrator for several people in the company. The need for the tool grew and the monolithic instance was used by 30+ teams having 500+ DAGs active resulting in 3.5 million tasks/month successfully finished.
At first, we moved to using a monolithic Airflow environment, but our needs quickly changed as we wanted to support a data mesh architecture within kiwi.com.
By leveraging Astronomer on GCP, we were able to move from a monolithic Airflow environment to many smaller instances of Airflow.


## Airflow as a Data Hybrid Cloud Orchestrator
[URL](https://airflowsummit.org/sessions/2023/airflow-as-a-data-hybrid-cloud-orchestrator/)  
Date: 2023-09-20T14:00:00-04:00
Summary:
Apache Airflow is Scalable, Dynamic, Extensible , Elegant and can it be a lot more ?
We have taken Airflow to the next level, using it as hybrid cloud data service accelerating our transformation.
During this talk we will present the implementation of Airflow as an orchestration solution between LEGACY, PRIVATE and PUBLIC cloud (AWS / AZURE) :
Comparison between public/private offers. Harness the power of Hybric cloud orchestrator to meet the regulatory requirements (European Financial Institutions) Real production use cases 

## Data Product DAGs
[URL](https://airflowsummit.org/sessions/2023/data-product-dags/)  
Date: 2023-09-20T14:00:00-04:00
Summary:
This talk will cover in high overview the architecture of a data product DAG, the benefits in a data mesh world and how to implement it easily.
Airflow is the de-facto orchestrator we use at Astrafy for all our data engineering projects. Over the years we have developed deep expertise in orchestrating data jobs and recently we have adopted the “data mesh” paradigm of having one Airlfow DAG per data product. Our standard data product DAGs contain the following stages:


## Deferrable Operators
[URL](https://airflowsummit.org/sessions/2023/deferrable-operators/)  
Date: 2023-09-20T14:00:00-04:00
Summary:
Deep dive into how AWS is developing Deferrable Operators for the Amazon Provider Package to help users realize the potential cost-savings provided by Deferrable Operators, and promote their usage.


## Eat, Sleep, Test, Repeat: How King ensures always-on data
[URL](https://airflowsummit.org/sessions/2023/eat-sleep-test-repeat-how-king-ensures-always-on-data/)  
Date: 2023-09-20T14:00:00-04:00
Summary:
At King, data is fundamental in helping us deliver the best possible experiences for the players of our games while continually bringing them new, innovative and evolving gameplay features. Data has to be “always-on”, where downtime and accuracy is treated with the same level of diligence as any of our games and success is measured against internal SLAs.
How is King using ‘data reliability engineering as code’ tools such as SodaCore within Airflow pipelines to detect, diagnose and inform about data issues to create coverage, improve quality & accuracy and help eliminate data downtime?


## Airflow at Monzo: Evolving our data platform as the bank scales
[URL](https://airflowsummit.org/sessions/2023/airflow-at-monzo-evolving-our-data-platform-as-the-bank-scales/)  
Date: 2023-09-20T14:30:00-04:00
Summary:
As a bank Monzo has seen exponential growth in active users, from 1.6 million in 2019 to 5.8 million in 2022. At the same time the number of data users and analysts has expanded from an initial team of 4 to 132. Alongside this growth, our infrastructure and tooling have had to evolve to deliver the same value at a new scale. From an Airflow installation deployed on a single monolithic instance we now deploy atop Kubernetes and have integrated our Airflow setup into the bank’s backend systems. This talk charts the story of that expansion and the growing pains we’ve faced, as well as looking to the future of our use of Airflow. We’ll first discuss how data at Monzo works, from event capture to arrival in our Data Warehouse, before assessing the challenges of our Airflow setup. We’ll then dive into the re-platforming that was required to meet our growing data needs, and some of the unique challenges that come with serving an ever growing user base and need for analysis and insight.


## Data at Rest: Bringing granular quality into flowing pipelines
[URL](https://airflowsummit.org/sessions/2023/data-at-rest-bringing-granular-quality-into-flowing-pipelines/)  
Date: 2023-09-20T14:30:00-04:00
Summary:
You’ve got your pipelines flowing … how much do you know about the data inside?
Most teams have some coverage with unit/contract/expectations tests, and you might have other quality checks. But it can be very ad-hoc and disorganized. You want to do more to beef up data quality and observability … does that mean you just need to write more tests and assertions?
Come learn about the best way to see your data’s quality alongside DAGs in a familiar context. We’ll review 3 common tools to get a handle on quality in a cohesive way across all your DAGs:


## Migrate Apache Oozie Workflows to Airflow and Run with Amazon EMR
[URL](https://airflowsummit.org/sessions/2023/migrate-apache-oozie-workflows-to-airflow-and-run-with-amazon-emr/)  
Date: 2023-09-20T14:30:00-04:00
Summary:
Learn how to convert Oozie Workflows into Airflow DAG and run it on Amazon EMR. The utility supports Airflow 2.4.3.
This utility is built on top of https://github.com/GoogleCloudPlatform/oozie-to-airflow


## OpenLineage in Airflow: A Comprehensive Guide
[URL](https://airflowsummit.org/sessions/2023/openlineage-in-airflow-a-comprehensive-guide/)  
Date: 2023-09-20T14:30:00-04:00
Summary:
With native support for OpenLineage in Airflow, users can now observe and manage their data pipelines with ease. This talk will cover the benefits of using OpenLineage, how it is implemented in Airflow, practical examples of how to take advantage of it, and what’s in our roadmap. Whether you’re an Airflow user or provider maintainer, this session will give you the knowledge to make the most of this tool.


## A New SQLAlchemyCollector and OpenLineageAdapter for Emitting Airflow Lineage Metadata as DAGs Run
[URL](https://airflowsummit.org/sessions/2023/a-new-sqlalchemycollector-and-openlineageadapter-for-emitting-airflow-lineage-metadata-as-dags-run/)  
Date: 2023-09-20T15:00:00-04:00
Summary:
Airflow uses SQLAlchemy under the hood but up to this point has not exploited the tool’s capacity to produce detailed metadata about queries, tables, columns, and more. In fact, SQLAlchemy ships with an event listener that, in conjunction with OpenLineage, offers tantalizing possibilities for enhancing the development process – specifically in the areas of monitoring and debugging.
SQLAlchemy’s event system features a Session object and ORMExecuteState mapped class that can be used to intercept statement executions and emit OpenLineage RunEvents as executions occur. In this talk, Michael Robinson from the community team at Astronomer will provide an overview and demo of new SQLAlchemyCollector and OpenLineageAdapter classes for leveraging SQLAlchemy’s event system to emit OpenLineage events as DAGs run.


## Airflow at UniCredit: Our journey from mainframe scheduling to modern data processing
[URL](https://airflowsummit.org/sessions/2023/airflow-at-unicredit-our-journey-from-mainframe-scheduling-to-modern-data-processing/)  
Date: 2023-09-20T15:00:00-04:00
Summary:
Representing the Murex Reporting team at UniCredit we would like to present our journey with Airflow, and how over the past two years it enabled us to automate and simplify our batch workflows. Comparing to our previous rigid mainframe scheduling approach, we have created a robust and scalable framework complete with a CI/CD process, bringing our time to market of scheduling changes down from 3 days to 1. Basing our solution on DAG networks joined by ResumeDagRunOperators and an array of custom-built plugins (such as static time predecessors) we were able to replicate the scheduling of our overnight ETL processes (consisting of approx. 8000 tasks with many-to-many dependencies) in Airflow, satisfying our bank reporting SLAs without performance regression and gaining massively improved process visibility and control. Our presentation will illustrate our journey and explore some of these customizations, which venture outside of Airflow’s core functionalities.


## Manifest destiny: Orchestrating dbt using Airflow
[URL](https://airflowsummit.org/sessions/2023/manifest-destiny-orchestrating-dbt-using-airflow/)  
Date: 2023-09-20T15:00:00-04:00
Summary:
Airflow is a popular choice for organizations looking to integrate open-source dbt within their existing data infrastructure. This talk will explore two primary methods of running dbt in Airflow: job-level and model-level. We’ll discuss the tradeoffs associated with each approach, highlighting the simplicity and efficiency of job-level orchestration, contrasted with the enhanced observability and control provided by model-level orchestration. We’ll also explain how the balance has shifted in recent years, with improvements to dbt core making model-level more efficient and innovative Airflow extensions like Cosmos making it easier to implement. Finally, we’ll provide benchmarks to help you determine which paradigm is the best fit for your needs.


## Micropipelines: A microservice approach for DAG authoring using datasets
[URL](https://airflowsummit.org/sessions/2023/micropipelines-a-microservice-approach-for-dag-authoring-using-datasets/)  
Date: 2023-09-20T15:00:00-04:00
Summary:
Introduced in Airflow 2.4, Datasets are a foundational feature for authoring modular data pipelines. As DAGs grow to encompass a larger number of data sources and encompass multiple data transformation steps, they typically become less predictable in the timeliness of execution and less efficient.
This talk focuses on leveraging Datasets to enable predictable and more efficient DAGs, by leveraging patterns from microservice architectures. Just as large monolithic applications were decomposed into micro-services to deliver more efficient scalability and faster development cycles, micropipelines have the same potential to radically transform data pipeline efficiency and development velocity.


## Airflow at Coinbase: How we supercharged the productivity of our users
[URL](https://airflowsummit.org/sessions/2023/airflow-at-coinbase-how-we-supercharged-the-productivity-of-our-users/)  
Date: 2023-09-20T15:30:00-04:00
Summary:
At Coinbase, Airflow is adopted by a wide range of applications, and used by nearly all the engineering and data science teams. In this session, we will share our journey in improving the productivity of Airflow users at Coinbase. The presentation will focus on three main topics:
Monorepo based architecture: our approach of using a monorepo to simplify DAG development and enable developers from across the company to work more efficiently and collaboratively.


## Elevating Data Quality: Great Expectations and Airflow at PepsiCo
[URL](https://airflowsummit.org/sessions/2023/elevating-data-quality-great-expectations/)  
Date: 2023-09-20T15:30:00-04:00
Summary:
Discover PepsiCo’s dynamic data quality strategy in a multi-cloud landscape. Join me, the Director of Data Engineering, as I unveil our Airflow utilization, custom operator integration, and the power of Great Expectations. Learn how we’ve harmonized Data Mesh into our decentralized development for seamless data integration. Explore our journey to maintain quality and enhance data as a strategic asset at PepsiCo.


## Platform for Genomic Processing Using Airflow and ECS
[URL](https://airflowsummit.org/sessions/2023/platform-for-genomic-processing-using-airflow-and-ecs/)  
Date: 2023-09-20T15:30:00-04:00
Summary:
High-scale orchestration of genomic algorithms using Airflow workflows, AWS Elastic Container Service (ECS), and Docker.
Genomic algorithms are highly demanding of CPU, RAM, and storage. Our data science team requires a platform to facilitate the development and validation of proprietary algorithms. The Data engineering team develops a research data platform that enables Data Scientists to publish docker images to AWS ECR and run them using Airflow DAGS that provision AWS’s ECS compute power of EC2 and Fargate.


## Traps and Misconceptions of Running Reliable Workloads in Apache Airflow
[URL](https://airflowsummit.org/sessions/2023/traps-and-misconceptions-of-running-reliable-workloads-in-apache-airflow/)  
Date: 2023-09-20T15:30:00-04:00
Summary:
Reliability is a complex and important topic. I will focus on both reliability definition and best practices.
I will begin by reviewing the Apache Airflow components that impact reliability. I will subsequently examine those aspects, showing the single points of failure, mitigations, and tradeoffs.
The journey starts with the scheduling process. I will focus on the aspects of Scheduler infrastructure and configuration that address reliability improvements. It doesn’t run in a vacuum therefore I’ll share my observations on the reliability aspect of Scheduler infrastructure.


## Airflow at Twitch: Our recommendation system starring Airflow
[URL](https://airflowsummit.org/sessions/2023/airflow-at-twitch-our-recommendation-system-starring-airflow/)  
Date: 2023-09-20T16:15:00-04:00
Summary:
Twitch, the world’s leading live streaming platform, has a massive user base of over 140 million active users and an incredibly complex recommendation system to deliver a personalized and engaging experience to its users.
In this talk, we will dive into how Twitch leverages the power of Apache Airflow to manage and orchestrate the training and deployment of its recommendation models. You will learn about the scale of Twitch’s reach and the challenges we faced in building a scalable, reliable, and developer-friendly recommendation system.


## From Pain Points to Best Practices: Enhancing Airflow migrations and local development at Wix.com
[URL](https://airflowsummit.org/sessions/2023/from-pain-points-to-best-practices-enhancing-airflow-migrations-and-local-development-at-wix-com/)  
Date: 2023-09-20T16:15:00-04:00
Summary:
Are you tired of spending hours on Airflow migrations and wondering how to make them more accessible? Would you like to be able to test your code on different Airflow versions? or are you struggling to set up a reliable local development environment?
These are some of the top pain points for data engineers working with Airflow. But fear not – Wix Data Engineering has some best practices to share that will make your life easier.


## Sketching Pipelines Using DAG Authoring UI
[URL](https://airflowsummit.org/sessions/2023/sketching-pipelines-using-dag-authoring-ui/)  
Date: 2023-09-20T16:15:00-04:00
Summary:
Cloudera Data Engineering (CDE) is a serverless service for Cloudera Data Platform that allows you to submit various Spark jobs and Airflow DAGs to an auto-scaling cluster.
Running your workloads as Python DAG files may be the usual, but not the most convenient way for some users as it involves a lot of background around syntaxes, the programming language, aesthetics of Airflow, etc.
The DAG Authoring UI is a tool built on top of Airflow APIs to allow one to use a graphical user interface to create, manage, and destroy complex DAGs. The DAG authoring UI will give one the ability to perform tasks on Airflow without really having to know DAG structure, Python programming language, and the internals of Airflow.


## Using Dynamic Task Mapping to Orchestrate dbt
[URL](https://airflowsummit.org/sessions/2023/using-dynamic-task-mapping-to-orchestrate-dbt/)  
Date: 2023-09-20T16:15:00-04:00
Summary:
Airflow, traditionally used by Data Engineers, is now popular among Analytics Engineers who aim to provide analysts with high-quality tooling while adhering to software engineering best practices. dbt, an open-source project that uses SQL to create data transformation pipelines, is one such tool. One approach to orchestrating dbt using Airflow is using dynamic task mapping to automatically create a task for each sub-directory inside dbt’s staging, intermediate, and marts directories. This enables analysts to write SQL code that is automatically added as a dedicated task in Airflow at runtime. Combining this new Airflow feature with dbt best practices offers several benefits, such as analysts not needing to make Airflow changes and engineers being able to re-run subsets of dbt models should errors occur. In this talk, I would like to share some lessons I have learned while successfully implementing this approach for several clients.


## Airflow at Gojek: Streamlining data processing for Tableau dashboards
[URL](https://airflowsummit.org/sessions/2023/airflow-at-gojek-streamlining-data-processing-for-tableau-dashboards/)  
Date: 2023-09-20T16:45:00-04:00
Summary:
With millions of orders per day, Gojek needs a data processing solution that can handle a high volume of data. Airflow is a scalable tool that can handle large volumes of data and complex workflows, making it an ideal solution for Gojek’s needs.
With Airflow, we can create automated data pipelines to extract data from various sources, transform it, and load it into dashboards such as Tableau for analysis and visualization. This eliminates the need for manual data transfers and reduces the risk of errors.


## Reducing Costs by Maximizing Airflow and DAG Performance
[URL](https://airflowsummit.org/sessions/2023/reducing-costs-by-maximizing-airflow-and-dag-performance/)  
Date: 2023-09-20T16:45:00-04:00
Summary:
Airflow DAGs are Python code (which can pretty much do anything you want) and Airflow has hundreds configuration options (which can dramatically change Airflow behavior). Those two facts contribute to endless combinations that can run the same workloads, but only a precious few are efficient. The rest will result in failed tasks and excessive compute usage, costing time and money.
This talk will demonstrate how small changes can yield big dividends, and reveals some code improvements and Airflow configurations that can reduce costs and maximize performance.


## Simplifying the Creation of Data Science Pipelines with Airflow
[URL](https://airflowsummit.org/sessions/2023/simplifying-the-creation-of-data-science-pipelines-with-airflow/)  
Date: 2023-09-20T16:45:00-04:00
Summary:
The ability to create DAGs programmatically opens up new possibilities for collaboration between Data Science and Data Engineering. Engineering and DevOPs are typically incentivized by stability whereas Data Science is typically incentivized by fast iteration and experimentation. With Airflow, it becomes possible for engineers to create tools that allow Data Scientists and Analysts to create robust no-code/low-code data pipelines for feature stores.
We will discuss Airlow as a means of bridging the gap between data infrastructure and modeling iteration as well as examine how a Qbiz customer did just this by creating a tool which allows Data Scientists to build features, train models and measure performance, using cloud services, in parallel.


## Supercharge Your Data Testing with a Fully Open Stack
[URL](https://airflowsummit.org/sessions/2023/supercharge-data-testing/)  
Date: 2023-09-20T16:45:00-04:00
Summary:
Are you tired of spending countless hours testing your data pipelines, only to find that they don’t work as expected? Do you wish there was a better way to manage your data versions and streamline your testing processes?
If so, this presentation is for you!
Join us as we explore the problem domain of testing environments for data pipelines and take a deep dive into the available tools currently in use. We’ll introduce you to the game-changing concepts of data versioning and lakeFS and show you how to integrate these tools with Airflow to revolutionize your testing workflows.


## Accelerating Data Delivery: How the FT automated its ETL pipelines with Airflow
[URL](https://airflowsummit.org/sessions/2023/accelerating-data-delivery-how-the-ft-automated-its-etl-pipelines-with-airflow/)  
Date: 2023-09-20T17:15:00-04:00
Summary:
Inside the Financial Times, we’ve been gradually moving our batching data processing from a custom solution to Airflow. To enable various teams within the company to use Airflow more effectively, we’ve been working on extending the system’s self-service capabilities. This includes giving ownership to teams of their DAGs and separating resources such as connections.
The batch data ingestion processes are the main ETL - like jobs that we run on Airflow. The creation of a new job used to be a manual and repetitive task of receiving the data specification, creating the requisite tables in our data warehouse and writing the DAG that would move the data there. Airflow allowed us to automate this process to a degree that surprised us, completely removing the need to write DAG code.


## Beyond Data Engineering: Airflow for Operations
[URL](https://airflowsummit.org/sessions/2023/beyond-data-engineering-airflow-operations/)  
Date: 2023-09-20T17:15:00-04:00
Summary:
Much of the world sees Airflow as a hammer and ETL tasks as nails, but in reality, Airflow is much more of a sophisticated multitool, capable of orchestrating a wide variety of complex workflows. Astronomer’s Customer Reliability Engineering (CRE) team is leveraging this potential in its development of Airline, a tool powered by Airflow that monitors Airflow deployments and sends alerts proactively when issues arise.
In this talk, Ryan Hatter from Astronomer will give an overview of Airline. He’ll explain how it integrates with ZenDesk, Kubernetes, and other services to resolve customers’ problems more quickly, and in many cases, even before customers realize there’s an issue.


## Chase The Sun: Build greener DAGs with VertFlow
[URL](https://airflowsummit.org/sessions/2023/chase-the-sun-build-greener-dags-with-vertflow/)  
Date: 2023-09-20T17:15:00-04:00
Summary:
In 2022, cloud data centres accounted for up to 3.7% of global greenhouse gas emissions, exceeding those of aviation and shipping.
Yet in the same year, Britain wasted 4 Terawatt hours of renewable energy because it couldn’t be transported from where it was generated to where it was needed.
So why not move the cloud to the clean energy?
VertFlow is an Airflow operator that deploys workloads to the greenest Google Cloud data centre, based on the realtime carbon intensity of electricity grids worldwide.


## Unlocking the Power of Warehouse Allocation: Optimizing task dispatching for cost efficiency
[URL](https://airflowsummit.org/sessions/2023/unlocking-the-power-of-warehouse-allocation-optimizing-task-dispatching-for-cost-efficiency/)  
Date: 2023-09-20T17:15:00-04:00
Summary:
In this session, we’ll explore the inner workings of our warehouse allocation service and its many benefits. We’ll discuss how you can integrate these principles into your own workflow and provide real-world examples of how this technology has improved our operations. From reducing queue times to making smart decisions about warehouse costs, warehouse allocation has helped us streamline our operations and drive growth.
With its seamless integration with Airflow, building an in-house warehouse allocation pipeline is simple and can easily fit into your existing workflow. Join us for this session to unlock the full potential of this in-house service and take your operations to the next level. Whether you’re a data engineer or a business owner, this technology can help you improve your bottom line and streamline your operations. Don’t miss out on this opportunity to learn more and optimize your workflow with the warehouse allocation service.


## Lightning talks and event wrap-up
[URL](https://airflowsummit.org/sessions/2023/lightning-talks/)  
Date: 2023-09-20T17:45:00-04:00
Summary:
We will have close Airflow Summit with lightning talks (5 minutes each).
You will be able to sign up during the event. We will only have space for 10 talks.


## Building an Airflow Pipeline with dbt and Snowflake
[URL](https://airflowsummit.org/sessions/2023/workshops/workshop-snowflake/)  
Date: 2023-09-21T09:00:00-04:00
Summary:
In this workshop you will learn how to simplify your data pipelines in the Snowflake Data Cloud.

## Hands-on Data Observability with Airflow + Modern Data Stack
[URL](https://airflowsummit.org/sessions/2023/workshops/databand-workshop/)  
Date: 2023-09-21T09:00:00-04:00
Summary:
Hands on workshop showing how data observability can work within your Airflow and Modern Data Stack.

## Lineage and Disaster Recovery procedures for Airflow based on Cloud Composer & Google Cloud Platform
[URL](https://airflowsummit.org/sessions/2023/workshops/cloud-composer-workshop/)  
Date: 2023-09-21T09:00:00-04:00
Summary:
Hands on workshop for medium/advanced Airflow users who would like to know more about Airflow and Composer and use features like data lineage to enhance observability and disaster recovery procedures.

## Performance optimize your Apache Airflow environment
[URL](https://airflowsummit.org/sessions/2023/workshops/performance-optimize-airflow-workshop/)  
Date: 2023-09-21T09:00:00-04:00
Summary:
Learn how to optimize your Apache Airflow environment. You will get hands-on experience implementing  techniques and best practices and see how they improve the performance of the Airflow environment.

## Airflow and Data Mesh: Architecture, setup & monitoring
[URL](https://airflowsummit.org/sessions/2023/workshops/data-mesh-workshop/)  
Date: 2023-09-21T12:00:00-04:00
Summary:
In this workshop you will learn why and how to set up a data mesh architecture based on Apache Airflow.

## Contributing to Apache Airflow
[URL](https://airflowsummit.org/sessions/2023/workshops/contribution-workshop/)  
Date: 2023-09-21T12:00:00-04:00
Summary:
Learn how you can become a contributor to Apache Airflow. From setting up an environment to making your first pull request.

## Developing Modern ETL Pipelines on Apache Airflow with Astro Python SDK
[URL](https://airflowsummit.org/sessions/2023/workshops/astro-python-sdk-developing-modern-etl-pipelines-on-apache-airflow/)  
Date: 2023-09-21T12:00:00-04:00
Summary:
The main objective of this workshop is to demonstrate how Apache Airflow, together with the Astro Python SDK, can be used to orchestrate data pipelines and perform ETL processes in a scalable and performant way for professionals.
During the workshop, you will learn how to easily create pipelines in Apache Airflow with the Astro Python SDK, fully open-source and accelerated by Astronomer company. Acquiring the skills from this workshop will enable you to implement data pipelines with few lines of code, using the best practices and recommendations in the market.


## Get Airflow Certified
[URL](https://airflowsummit.org/sessions/2023/workshops/get-airflow-certified/)  
Date: 2023-09-21T12:00:00-04:00
Summary:
During Airflow Summit you can take an Airflow Certification exam at no additional cost. We will have beginner and advanced level certifications available.

## 10 years of Airflow: history, insights, and looking forward
[URL](https://airflowsummit.org/sessions/2024/10-years-of-airflow-history-insights-and-looking-forward/)  
Date: 2024-09-10 09:10:00
Summary:
10 years after its creation, Airflow is stronger than ever: in last year’s Airflow survey, 81% of users said Airflow is important or very important to their business, 87% said their Airflow usage has grown over time, and 92% said they would recommend Airflow. In this panel discussion, we’ll celebrate a decade of Airflow and delve into how it became the highly recommended industry standard it is today, including history, pivotal moments, and the role of the community. Our panel of seasoned experts will also talk about where Airflow is going next, including future use cases like generative AI and the highly anticipated Airflow 3.0. Don’t miss this insightful exploration into one of the most influential tools in the data landscape.


## Security United: collaborative effort on securing Airflow ecosystem with Alpha-Omega, PSF & ASF
[URL](https://airflowsummit.org/sessions/2024/security-united-collaborative-effort-on-securing-airflow-ecosystem-with-alpha-omega-psf-asf/)  
Date: 2024-09-10 10:05:00
Summary:
Airflow’s power comes from its vast ecosystem, but securing this intricate web requires a united front. This talk unveils a groundbreaking collaborative effort between the Python Software Foundation (PSF), the Apache Software Foundation (ASF), the Airflow Project Management Committee (PMC), and Alpha-Omega Fund - aimed at securing not only Airflow, but the whole ecosystem. We’ll explore this new project dedicated to improving security across the Airflow landscape.


## Airflow at Burns & McDonnell | Orchestration from zero to 100
[URL](https://airflowsummit.org/sessions/2024/airflow-at-burns-mcdonnell-orchestration-from-zero-to-100/)  
Date: 2024-09-10 11:00:00
Summary:
As the largest employee-owned engineering and construction firm in the United States, Burns & McDonnell has a massive amount of data. Not only that, it’s hard to pinpoint which source system has the data we need. Our solution to this challenge is to build a unified information platform — a single source of truth where all of our data is searchable, trustworthy, and accessible to our employee-owners and the projects that need it.


## Airflow UI Roadmap
[URL](https://airflowsummit.org/sessions/2024/airflow-ui-roadmap/)  
Date: 2024-09-10 11:00:00
Summary:
Soon we will finally switch to a 100% React UI with a full separation between the API and UI as well. While we are doing such a big change, let’s also take the opportunity to imagine whole new interfaces vs just simply modernizing the existing views. How can we use design to help you better understand what is going on with your DAG?
Come listen to some of our proposed ideas and bring your own big ideas as the second half will be an open discussion.


## Building on Cosmos: Making dbt on Airflow Easy
[URL](https://airflowsummit.org/sessions/2024/building-on-cosmos-making-dbt-on-airflow-easy/)  
Date: 2024-09-10 11:00:00
Summary:
Balyasny Asset Management (BAM) is a diversified global investment firm founded in 2001 with over $20 billion in assets under management.
As dbt took hold at BAM, we had multiple teams building dbt projects against Snowflake, Redshift, and SQL Server. The common question was: How can we quickly and easily productionise our projects?
Airflow is the orchestrator of choice at BAM, but our dbt users ranged from Airflow power users to people who’d never heard of Airflow before. We built a single solution on top of Cosmos that allowed us to:


## Event-driven Data Pipelines with Apache Airflow
[URL](https://airflowsummit.org/sessions/2024/event-driven-data-pipelines-with-apache-airflow/)  
Date: 2024-09-10 11:00:00
Summary:
Airflow is all about schedules…we use CRON strings and Timetable to define schedules, and there’s an Airflow Scheduler component that manages those timetables, and a lot more, to ensure that DAGs and tasks are addressed based on those schedules.
But what do you do if your data isn’t available on a schedule? What if data is coming from many sources, at varying times, and your job is to make sure it’s all as up-to-date as possible? An event-driven data pipeline may be the answer.


## Activating operational metadata with Airflow, Atlan and OpenLineage
[URL](https://airflowsummit.org/sessions/2024/activating-operational-metadata-with-airflow-atlan-and-openlineage/)  
Date: 2024-09-10 12:00:00
Summary:
OpenLineage is an open standard for lineage data collection, integrated into the Airflow codebase, facilitating lineage collection across providers like Google, Amazon, and more.
Atlan Data Catalog is a 3rd generation active metadata platform that is a single source of trust unifying cataloging, data discovery, lineage, and governance experience.
We will demonstrate what OpenLineage is and how, with minimal and intuitive setup across Airlfow and Atlan, it presents unified workflows view, efficient cross-platform lineage collection, including column level, in various technologies (Python, Spark, dbt, SQL etc.) and clouds (AWS, Azure, GCP, etc.) - all orchestrated by Airflow.


## Orchestrating & Optimizing a Batch Ingestion Data Platform for Americas #1 Sportsbook
[URL](https://airflowsummit.org/sessions/2024/orchestrating-optimizing-a-batch-ingestion-data-platform-for-americas-1-sportsbook/)  
Date: 2024-09-10 12:00:00
Summary:
FanDuel Group, an industry leader in sports-tech entertainment, is proud to be recognized as the #1 sports betting company in the US as of 2023 with 53.4% market share. With a workforce exceeding 4,000 employees, including over 100 data engineers, FanDuel Group is at the forefront of innovation in batch processing orchestration platforms. Currently, our platform handles over 250,000 DAG runs & executes ~3 million tasks monthly across 17 deployments. It provides a standardized framework for pipeline development, structured observability, monitoring, & alerting. It also offers automated data processing managed by an in-house team, enabling stakeholders to concentrate on core business objectives. Our batch ingestion platform is the backbone of endless use cases, facilitating the landing of data into storage at scheduled intervals, real-time ingestion of micro batches triggered by events, standardization processes, & ensuring data availability for downstream applications. Our proposed session also delves into our forward-looking tech strategy as well as addressing the expansion of orchestration diversity by integrating scheduled jobs from various domains into our robust data platform.


## Running Airflow Tasks Anywhere, in any Language
[URL](https://airflowsummit.org/sessions/2024/running-airflow-tasks-anywhere-in-any-language/)  
Date: 2024-09-10 12:00:00
Summary:
Imagine a world where writing Airflow tasks in languages like Go, R, Julia, or maybe even Rust is not just a dream but a native capability. Say goodbye to BashOperators; welcome to the future of Airflow task execution.
Here’s what you can expect to learn from this session:
Multilingual Tasks: Explore how we empower DAG authors to write tasks in any language while retaining seamless access to Airflow Variables and Connections. Simplified Development and Testing: Discover how a standardized interface for task execution promises to streamline development efforts and elevate code maintainability. Enhanced Scalability and Remote Workers: Learn how enabling tasks to run on remote workers opens up possibilities for seamless deployment on diverse platforms, including Windows and remote Spark or Ray clusters. Experience the convenience of effortless deployments as we unlock new avenues for Airflow usage. Join us as we embark on an exploratory journey to shape the future of Airflow task execution. Your insights and contributions are invaluable as we refine this vision together. Let’s chart a course towards a more versatile, efficient, and accessible Airflow ecosystem.


## Unleash the Power of AI: Streamlining Airflow DAG Development with AI-Driven Automation
[URL](https://airflowsummit.org/sessions/2024/unleash-the-power-of-ai-streamlining-airflow-dag-development-with-ai-driven-automation/)  
Date: 2024-09-10 12:00:00
Summary:
Nowadays, conversational AI is no longer exclusive to large enterprises. It has become more accessible and affordable, opening up new possibilities and business opportunities. In this session, discover how you can leverage Generative AI as your AI pair programmer to suggest DAG code and recommend entire functions in real-time, directly from your editor. Visualize how to harness the power of ML, trained on billions of lines of code, to transform natural language prompts into coding suggestions. Seamlessly cycle through lines of code, complete function suggestions, and choose to accept, reject, or edit them. Witness firsthand how Generative AI provides recommendations based on the project’s context and style conventions. The objective is to equip you with techniques that allow you to spend less time on boilerplate and repetitive code patterns, and more time on what truly matters: building exceptional orchestration software.


## dbt-Core & Airflow 101: Building Data Pipelines Demystified
[URL](https://airflowsummit.org/sessions/2024/dbt-core-airflow-101-building-data-pipelines-demystified/)  
Date: 2024-09-10 14:00:00
Summary:
dbt became the de facto for data teams building reliable and trustworthy SQL code leveraging a modern data stack architecture. The dbt logic needs to be orchestrated, and jobs scheduled to meet business expectations. That’s where Airflow comes into play.
In this quick introduction session, you’ll gonna learn:
How to leverage dbt-Core & Airflow to orchestrate pipelines
Write DAGs in a Pythonic way
Apply best practices on your jobs


## Optimize Your DAGs: Embrace Dag Params for Efficiency and Simplicity
[URL](https://airflowsummit.org/sessions/2024/optimize-your-dags-embrace-dag-params-for-efficiency-and-simplicity/)  
Date: 2024-09-10 14:00:00
Summary:
In the realm of data engineering, there is a prevalent tendency for professionals to develop similar Directed Acyclic Graphs (DAGs) to manage analogous tasks. Leveraging Dag Params presents an effective strategy for mitigating redundancy within these DAGs. Moreover, the utilization of Dag Params facilitates seamless enforcement of user inputs, thereby streamlining the process of incorporating validations into the DAG codebase.


## Streamlining a Mortgage ETL Pipeline with Apache Airflow
[URL](https://airflowsummit.org/sessions/2024/streamlining-a-mortgage-etl-pipeline-with-apache-airflow/)  
Date: 2024-09-10 14:00:00
Summary:
At Bloomberg, it is our team’s responsibility to ensure the timely delivery to our clients worldwide of a vast dataset comprising approximately 5 billion data points on roughly 50 million loans and over 1.4 million securities, disclosed twice a month by three major government-sponsored mortgage entities. Ingesting this data so we can create and derive complex data structures to be consumed by our applications for our clients has been our biggest challenge. In this talk, we will discuss our transition from a manually-managed spreadsheet-based system to an automated centralized orchestration tool, and how Apache Airflow has helped make the process more transparent, predictable, and visible.


## Unlocking FMOps/LLMOps with Airflow: A guide to operationalizing and managing Large Language Models
[URL](https://airflowsummit.org/sessions/2024/unlocking-fmops-llmops-using-apache-airflow-a-guide-to-operationalizing-and-managing-large-language/)  
Date: 2024-09-10 14:00:00
Summary:
In the last few years Large Language Models (LLMs) have risen to prominence as outstanding tools capable of transforming businesses. However, bringing such solutions and models to the business-as-usual operations is not an easy task. In this session, we delve into the operationalization of generative AI applications using MLOps principles, leading to the introduction of foundation model operations (FMOps) or LLM operations using Apache Airflow. We further zoom into aspects of expected people and process mindsets, new techniques for model selection and evaluation, data privacy, and model deployment. Additionally, know how you can use the prescriptive features of Apache Airflow to aid your operational journey. Whether you are building using out of the box models (open-source or proprietary), creating new foundation models from scratch, or fine-tuning an existing model, with the structured approaches described you can effectively integrate LLMs into your operations, enhancing efficiency and productivity without causing disruptions in the cloud or on-premises.


## Data Orchestration for Emerging Technology Analysis
[URL](https://airflowsummit.org/sessions/2024/data-orchestration-for-emerging-technology-analysis/)  
Date: 2024-09-10 14:35:00
Summary:
The Center for Security and Emerging Technology is a think tank at Georgetown University that studies security implications of emerging technologies, including data-driven analyses across bibliometric, patenting, and investment datasets. This talk will describe CSET’s data infrastructure which uses Airflow to orchestrate data ingestion, model deployment, webscraping, and manual data curation pipelines. We’ll also discuss how outputs from these pipelines are integrated into public-facing web applications and written reports, and some lessons learned from building and maintaining data pipelines on a data team with a diverse skill set.


## From Oops to Ops: Smart Task Failure Diagnosis with OpenAI
[URL](https://airflowsummit.org/sessions/2024/from-oops-to-ops-smart-task-failure-diagnosis-with-openai/)  
Date: 2024-09-10 14:35:00
Summary:
This session reveals an experimental venture integrating OpenAI’s AI technologies with Airflow, aimed at advancing error diagnosis.
Through the application of AI, our objective is to deepen the understanding of issues, provide comprehensive insights into task failures, and suggest actionable solutions, thereby augmenting the resolution process. This method seeks to not only enhance diagnostic efficiency but also to equip data engineers with AI-informed recommendations.
Participants will be guided through the integration journey, illustrating how AI can refine error analysis and potentially simplify troubleshooting workflows.


## From Tech Specs to Business Impact: How to Design A Truly End-to-End Airflow Project
[URL](https://airflowsummit.org/sessions/2024/from-tech-specs-to-business-impact-how-to-design-a-truly-end-to-end-airflow-project/)  
Date: 2024-09-10 14:35:00
Summary:
There are many Airflow tutorials. However, many don’t show the full process of sourcing, transforming, testing, alerting, documenting, and finally supplying data. This talk with go over how to piece together an end-to-end Airflow project that transforms raw data to be consumable by the business. It will include how various technologies can all be orchestrated by Airflow to satisfy the needs of analysts, engineers, and business stakeholders.
The talk will be divided into the following sections:


## Managing version upgrades without feelings of terror
[URL](https://airflowsummit.org/sessions/2024/managing-version-upgrades-without-feelings-of-terror/)  
Date: 2024-09-10 14:35:00
Summary:
Airflow version upgrades can be challenging. Maybe you upgrade and your dags fail to parse (that’s an easy fix). Or maybe you upgrade and everything looks fine, but when your dag runs, you can no longer connect to mysql because the TLS version changed. In this talk I will provide concrete strategies that users can put into practice to make version upgrades safer and less painful. Topics may include:
What semver means and what it implies for the upgrade process


## A deep dive into Airflow configuration options for scalability
[URL](https://airflowsummit.org/sessions/2024/a-deep-dive-into-airflow-configuration-options-for-scalability/)  
Date: 2024-09-10 15:10:00
Summary:
Apache Airflow has a lot of configuration options. A change in some of these options can affect the performance of Airflow.
If you are wondering why your Airflow instance is not running the number of tasks you expected it to run, after this talk, you will have a better understanding of the configuration options available for improving the number of tasks your Airflow instance can run.
We will talk about the DAG parsing configuration options, options for scheduler scalability, etc., and the pros and cons of these options.


## Airflow Datasets and Pub/Sub for Dynamic DAG Triggering
[URL](https://airflowsummit.org/sessions/2024/airflow-datasets-and-pub-sub-for-dynamic-dag-triggering/)  
Date: 2024-09-10 15:10:00
Summary:
Looking for a way to streamline your data workflows and master the art of orchestration? As we navigate the complexities of modern data engineering, Airflow’s dynamic workflow and complex data pipeline dependencies are starting to become more and more common nowadays. In order to empower data engineers to exploit Airflow as the main orchestrator, Airflow Datasets can be easily integrated in your data journey.
This session will showcase the Dynamic Workflow orchestration in Airflow and how to manage multi-DAGs dependencies with Multi-Dataset listening.


## Elevating Machine Learning Deployment: Unleashing the Power of Airflow in Wix's ML Platform
[URL](https://airflowsummit.org/sessions/2024/elevating-machine-learning-deployment-unleashing-the-power-of-airflow-in-wix-s-ml-platform/)  
Date: 2024-09-10 15:10:00
Summary:
In his presentation, Elad will provide a novel take on Airflow, highlighting its versatility beyond conventional use for scheduled pipelines. He’ll discuss its application as an on-demand tool for initiating and halting jobs, mainly in the Data Science fields, like dataset enrichment and batch prediction via API calls, complete with real-time status tracking and alerts. The talk aims to encourage a fresh approach to Airflow utilization but will also delve into the technical aspects of implementing DAG triggering and cancellation logic.


## How Panasonic Leverages Airflow
[URL](https://airflowsummit.org/sessions/2024/how-panasonic-leverages-airflow/)  
Date: 2024-09-10 15:10:00
Summary:
Using various operators to perform daily routines.
Integration with Technologies:
Redis: Acts as a caching mechanism to optimize data retrieval and processing speed, enhancing overall pipeline performance.
MySQL: Utilized for storing metadata and managing task state information within Airflow’s backend database.
Tableau: Integrates with Airflow to generate interactive visualizations and dashboards, providing valuable insights into the processed data.
Amazon Redshift: Panasonic leverages Redshift for scalable data warehousing, seamlessly integrating it with Airflow for data loading and analytics.


## Airflow at NCR Voyix: Streamlining ML workflows development with Airflow
[URL](https://airflowsummit.org/sessions/2024/airflow-at-ncr-voyix-streamlining-ml-workflows-development-with-airflow/)  
Date: 2024-09-10 16:00:00
Summary:
NCR Voyix Retail Analytics AI team offers ML products for retailers while embracing Airflow as its MLOps Platform. As the team is small and there have been twice as many data scientists as engineers, we encountered challenges in making Airflow accessible to the scientists:
As they come from diverse programming backgrounds, we needed an architecture enabling them to develop production-ready ML workflows without prior knowledge of Airflow.
Due to dynamic product demands, we had to implement a mechanism to interchange Airflow operators effortlessly.


## Empowering More Teams in your Organization to Self-service their Airflow Needs
[URL](https://airflowsummit.org/sessions/2024/empowering-more-teams-in-your-organization-to-self-service-their-airflow-needs/)  
Date: 2024-09-10 16:00:00
Summary:
Does your organization feel like the responsibility to write Airflow DAGs, handle the Airflow infrastructure administration, debug failing tasks, and keep up with new features and best practices is too much for too few people? Perhaps you only have one data team that owns all of that; or you have too many teams that have too many permissions into other teams’ DAGs.
The topic of this talk is how Rakuten Kobo enables self-service for various teams within its organization to build their own DAGs in Airflow. The talk will include how we delineate the Airflow responsibilities of various teams, build guard rails for new Airflow developers, how different teams automatically have permissions required for their “own” DAGs (but not others), the unique responsibilities of Operations and Data Engineering teams, and how it is done in a scalable manner.


## Optimizing Airflow Performance: Strategies, Techniques, and Best Practices
[URL](https://airflowsummit.org/sessions/2024/optimizing-airflow-performance-strategies-techniques-and-best-practices/)  
Date: 2024-09-10 16:00:00
Summary:
Airflow, an open-source platform for orchestrating complex data workflows, is widely adopted for its flexibility and scalability. However, as workflows grow in complexity and scale, optimizing Airflow performance becomes crucial for efficient execution and resource utilization. This session delves into the importance of optimizing Airflow performance and provides strategies, techniques, and best practices to enhance workflow execution speed, reduce resource consumption, and improve system efficiency. Attendees will gain insights into identifying performance bottlenecks, fine-tuning workflow configurations, leveraging advanced features, and implementing optimization strategies to maximize pipeline throughput. Whether you’re a seasoned Airflow user or just getting started, this session equips you with the knowledge and tools needed to optimize your Airflow deployments for optimal performance and scalability. We’ll also explore topics such as DAG writing best practices, monitoring and updating Airflow configurations, and database performance optimization, covering unused indexes, missing indexes, and minimizing table and index bloat.


## Streamline data science workflow development using Jupyter notebooks and Airflow
[URL](https://airflowsummit.org/sessions/2024/streamline-data-science-workflow-development-using-jupyter-notebooks-and-airflow/)  
Date: 2024-09-10 16:00:00
Summary:
Jupyter Notebooks are widely used by data scientists and engineers to prototype and experiment with data. However these engineers are often required to work with other data or platform engineers to productionize these experiments due to the complexity in navigating infrastructure and systems. In this talk, we will deep dive into this PR https://github.com/apache/airflow/pull/34840 and share how airflow can be leveraged as a platform to execute notebook pipelines (python, scala or spark) in dynamic environments like Kubernetes for various heterogeneous use cases. We will demonstrate how data scientists can use a Jupyter extension to easily build and manage such pipelines which are executed using Airflow streamlining data science workflow development and supercharging productivity


## Automated Testing and Deployment of DAGs
[URL](https://airflowsummit.org/sessions/2024/automated-testing-and-deployment-of-dags/)  
Date: 2024-09-10 16:35:00
Summary:
DAG integrity is critical. So are coding conventions, consistency in standards for the group. In this talk, we will share the various lessons learned for testing/verifying our DAGs as part of our GitHub workflows [ for testing as part of the pull request process, and for automated deployment - eventually to production - once merged ]. We will dig into how we have unlocked additional efficiencies, catch errors before they get deployed, and generally how we are better off for having both Airflow & plenty of checks in our CI, before we merge/deploy.


## Evolution of Orchestration at GoDaddy: A Journey from On-prem to Cloud-based Single Pane Model
[URL](https://airflowsummit.org/sessions/2024/evolution-of-orchestration-at-godaddy-a-journey-from-on-prem-to-cloud-based-single-pane-model/)  
Date: 2024-09-10 16:35:00
Summary:
Explore the evolutionary journey of orchestration within GoDaddy, tracing its transformation from initial on-premise deployment to a robust cloud-based Apache Airflow orchestration model. This session will detail the pivotal shifts in design, organizational decisions, and governance that have streamlined GoDaddy’s Data Platform and enhanced overall governance.
Attendees will gain insights valuable for optimizing Airflow deployments and simplifying complex orchestration processes.
Recap of the transformation journey and its impact on GoDaddy’s data operations.


## Orchestration of ML workloads via Airflow & GKE Batch
[URL](https://airflowsummit.org/sessions/2024/orchestration-of-ml-workloads-via-airflow-gke-batch/)  
Date: 2024-09-10 16:35:00
Summary:
During this talk we are going to given an overview of different orchestration approaches (Kubeflow, Ray, Airflow, etc.) when running ML workloads on Kubernetes and specifically we will focus on how to use Kubernetes Batch API and Kubernetes Operators to run complex ML workloads.


## Scaling AI Workloads with Apache Airflow
[URL](https://airflowsummit.org/sessions/2024/scaling-ai-workloads-with-apache-airflow/)  
Date: 2024-09-10 16:35:00
Summary:
AI workloads are becoming increasingly complex, with unique requirements around data management, compute scalability, and model lifecycle management. In this session, we will explore the real-world challenges users face when operating AI at scale. Through real-world examples, we will uncover common pitfalls in areas like data versioning, reproducibility, model deployment, and monitoring. Our practical guide will highlight strategies for building robust and scalable AI platforms leveraging Airflow as the orchestration layer and AWS for its extensive AI/ML capabilities. We will showcase how users have tackled these challenges, streamlined their AI workflows, and unlocked new levels of productivity and innovation.


## AI Reality Checkpoint: The Good, the Bad, and the Overhyped
[URL](https://airflowsummit.org/sessions/2024/ai-reality-checkpoint-the-good-the-bad-and-the-overhyped/)  
Date: 2024-09-10 17:10:00
Summary:
In the past 18 months, artificial intelligence has not just entered our workspaces – it has taken over. As we stand at the crossroads of innovation and automation, it’s time for a candid reflection on how AI has reshaped our professional lives, and to talk about where it’s been a game changer, where it’s falling short, and what’s about to shift dramatically in the short term.
Since the release of ChatGPT in December 2022, I’ve developed a “first-reflex” to augment and accelerate nearly every task with AI. As a founder and CEO, this spans a wide array of responsibilities from fundraising, internal communications, legal, operations, product marketing, finance, and beyond. In this keynote, I’ll cover diverse use cases across all areas of business, offering a comprehensive view of AI’s impact.


## Optimizing Critical Operations: Enhancing Robinhood's Workflow Journey with Airflow
[URL](https://airflowsummit.org/sessions/2024/optimizing-critical-operations-enhancing-robinhood-s-workflow-journey-with-airflow/)  
Date: 2024-09-10 17:10:00
Summary:
Airflow is widely used within Robinhood. In addition to traditional offline analytics use cases (to schedule ingestion and analytics workloads that populate our data lake), we also use Airflow in our backend services to orchestrate various workflows that are highly critical for the business, e.g: compliance and regulatory reporting, user facing reports and more.
As part of this, we have evolved what we believe is a unique deployment architecture for Airflow. We have central schedulers that are responsible for workloads from multiple different teams, but the workflow tasks themselves run on workers owned by respective teams that are highly coupled with their backend services and codebase.


## Overcoming Custom Python Package Hurdles in Airflow
[URL](https://airflowsummit.org/sessions/2024/overcoming-custom-python-package-hurdles-in-airflow/)  
Date: 2024-09-10 17:10:00
Summary:
DAG Authors, while constructing DAGs, generally use native libraries provided by Airflow in conjunction with python libraries available over public PyPI repositories.
But sometimes, DAG authors need to construct DAG using libraries that are either in-house or not available over public PyPI repositories. This poses a serious challenge for users who want to run their custom code with Airflow DAGs, particularly when Airflow is deployed in a cloud-native fashion.
Traditionally, these packages are baked in Airflow Docker images. This won’t work post deployment and is super impractical if your library is under development.


## Streamlining DAG Creation with YAML in Large Organizations
[URL](https://airflowsummit.org/sessions/2024/streamlining-dag-creation-yaml/)  
Date: 2024-09-10 17:10:00
Summary:
As organizations grow, the task of creating and managing Airflow DAGs efficiently becomes a challenge.
In this talk, we will delve into innovative approaches to streamlining Airflow DAG creation using YAML. By leveraging YAML configuration, we allow users to dynamically generate Airflow DAGs without requiring Python expertise or deep knowledge of Airflow primitives.
We will showcase the significant benefits of this approach, including eliminating duplicate configurations, simplifying DAG management for a large group of workflows, and ultimately enhancing productivity within large organizations.


## LLMs for Software Development & Apache Airflow
[URL](https://airflowsummit.org/sessions/2024/deepmind/)  
Date: 2024-09-11 09:00:00
Summary:
Artificial Intelligence is reshaping the landscape of software development. In this talk, we’ll explore the latest AI breakthroughs improving LLM capabilities for software development use cases.
We’ll discuss work and ideas in the field related to Airflow, particularly around model capabilities related to Python, DSLs, and low-resource languages.


## Winning Strategies: Powering a World Series Victory with Airflow Orchestration
[URL](https://airflowsummit.org/sessions/2024/winning-strategies-powering-a-world-series-victory-with-airflow-orchestration/)  
Date: 2024-09-11 09:30:00
Summary:
Dive into the winning playbook of the 2023 World Series Champions Texas Rangers, and discover how they leverage Apache Airflow to streamline their data pipelines. In this session, we’ll explore how real-world data pipelines enable agile decision-making and drive competitive advantage in the high-stakes world of professional baseball, all by using Airflow as an orchestration platform. Whether you’re a seasoned data engineer or just starting out, this session promises actionable strategies to elevate your data orchestration game to championship levels.


## Airflow as a workflow for Self Service Based Ingestion
[URL](https://airflowsummit.org/sessions/2024/airflow-as-a-workflow-for-self-service-based-ingestion/)  
Date: 2024-09-11 10:30:00
Summary:
Our Idea to platformize Ingestion pipelines is driven via Airflow in the background and streamline the entire ingestion process for Self Service.
With customer experience on top of it and making data ingestion fool proof as part of Analytics data team, Airflow is just complementing for our vision.


## Airflow at Ford: A Job Router Training Advance Driver Assistance Systems
[URL](https://airflowsummit.org/sessions/2024/airflow-at-ford-a-job-router-training-advance-driver-assistance-systems/)  
Date: 2024-09-11 10:30:00
Summary:
Ford Motor Company operates extensively across various nations. The Data Operations (DataOps) team for Advanced Driver Assistance Systems (ADAS) at Ford is tasked with the processing of terabyte-scale daily data from lidar, radar, and video. To manage this, the DataOps team is challenged with orchestrating diverse, compute-intensive pipelines across both on-premises infrastructure and the GCP and deal with sensitive of customer data across both environments The team is also responsible for facilitating the execution of on-demand, compute-intensive algorithms at scale through. To achieve these objectives, the team employs Astronomer/Airflow at the core of its strategic approach. This involves various deployments of Astronomer/Airflow that integrate seamlessly and securely (via Apigee) to initiate batch data processing and ML jobs on the cloud, as well as compute-intensive computer vision tasks on-premises, with essential alerting provided through the ELK stack. This presentation will delve into the architecture and strategic planning surrounding the hybrid batch router, highlighting its pivotal role in promoting rapid innovation and scalability in the development of ADAS features.


## Growing with Apache Airflow: A Providers Journey
[URL](https://airflowsummit.org/sessions/2024/growing-with-apache-airflow-a-providers-journey/)  
Date: 2024-09-11 10:30:00
Summary:
It has been nearly 4 years since the launch of Managed Workflows for Apache Airflow (MWAA) by AWS. It has gone through the trials and tribulations as with any new idea, working with customers to better understand its shortcomings, building dedicated teams focused on scaling and growth, and at its core, preserving the integrity and functionality of Apache Airflow. Initially launched with Airflow 1.10, MWAA is now available globally in multiple AWS regions supporting the latest version of Airflow along with a multitude of features. In this talk, we will cover a bit of that history along with debunking a few myths surrounding the critical needs for users today. From compliance requirements, larger environments, observability, and pricing, we will discuss how MWAA has evolved and continues to grow through its focus on customer value and more importantly, its dedication to the Apache Airflow community.


## Integrating dbt with Airflow: Overcoming performance hurdles
[URL](https://airflowsummit.org/sessions/2024/overcoming-performance-hurdles-in-integrating-dbt-with-airflow/)  
Date: 2024-09-11 10:30:00
Summary:
The integration between dbt and Airflow is a popular topic in the community, both in previous editions of Airflow Summit, in Coalesce and the #airflow-dbt Slack channel.
Astronomer Cosmos (https://github.com/astronomer/astronomer-cosmos/) stands out as one of the libraries that strives to enhance this integration, having over 300k downloads per month.
During its development, we’ve encountered various performance challenges in terms of scheduling and task execution. While we’ve managed to address some, others remain to be resolved.


## Behaviour Driven Development in Airflow
[URL](https://airflowsummit.org/sessions/2024/behaviour-driven-development-in-airflow/)  
Date: 2024-09-11 11:00:00
Summary:
Behaviour Driven Development can, in the simplest of terms, be described as Test Driven Development, only readable. It is of course more than that, but that is not the aim of this talk. This talk aims to show:
How to write tests before you write a single line of Airflow code
Create reusable and readable steps for setting up tests, in a given-when-then manner.
Test rendering and execution of your DAG’s tasks


## Boost Airflow Monitoring and Alerting with Automation Analytics & Intelligence by Broadcom
[URL](https://airflowsummit.org/sessions/2024/boost-airflow-monitoring-and-alerting-with-automation-analytics-intelligence-by-broadcom/)  
Date: 2024-09-11 11:00:00
Summary:
This talk is presented by Broadcom.
Airflow’s “workflow as code” approach has many benefits, including enabling dynamic pipeline generation and flexibility and extensibility in a seamless development environment. However, what challenges do you face as you expand your Airflow footprint in your organization? What if you could enhance Airflow’s monitoring capabilities, forecast DAG and task executions, obtain predictive alerting, visualize trends, and get more robust logging?
Broadcom’s Automation Analytics & Intelligence (AAI) offers advanced analytics for workload automation for cloud and on-premises automation. It connects easily with Airflow to offer improved visibility into dependencies between tasks in Airflow DAGs along with the workload’s critical path, dynamic SLA management, and more.


## A New DAG Paradigm: Less Airflow more DAGs
[URL](https://airflowsummit.org/sessions/2024/a-new-dag-paradigm-less-airflow-more-dags/)  
Date: 2024-09-11 11:30:00
Summary:
Astronomer’s data team recently underwent a major shift in how we work with Airflow. We’ll deep dive into the challenges which prompted that change, how we addressed them and where we are now.
This re-architecture included:
Switching to dataset scheduling and micro-pipelines to minimize failures and increase reliability.
Implementing a Control DAG for complex dependency management and full end-to-end pipeline visibility.
Standardized Task Groups for quick onboarding and scalability.
With Airflow managing itself, we can once again focus on the data rather than the operational overhead. As proof we’ll share our favorite statistics from the terabyte of data we process daily revealing insights into how the world’s data teams use Airflow.


## Comparing Airflow Executors and Custom Environments
[URL](https://airflowsummit.org/sessions/2024/comparing-airflow-executors-and-custom-environments/)  
Date: 2024-09-11 11:30:00
Summary:
With recent works in the direction of Executor Decoupling and interest in Hybrid Execution, we find it’s still quite common for Airflow users to use the old-time rule of thumbs like “Don’t Use Airflow with LocalExecutor in production”, “If your scheduler lags, split your DAGs over two separate Airflow Clusters”, and so on. In our talk, we will show a deep dive comparison between various Execution models Airflow support and hopefully update understanding of their efficiency and limitations.


## Linkedin's Continuous Deployment
[URL](https://airflowsummit.org/sessions/2024/linkedin-s-continuous-deployment/)  
Date: 2024-09-11 11:30:00
Summary:
LinkedIn Continuous Deployment (LCD), started with the goal of improving the deployment experience and expanding its outreach to all LinkedIn systems. LCD delivers a modern deployment UX and easy-to-customize pipelines which enables all LinkedIn applications to declare their deployment pipelines.
LCD’s vision is to automate cluster provisioning, deployments and enable touchless (continuous) deployments while reducing the manual toil involved in deployments.
LCD is powered by Airflow to orchestrate its deployment pipelines and automate the validation steps. For our customers Airflow is an implementation detail and we have well abstracted it out with our no-code/low code pipelines. Users describe their pipeline intent (via CLI/UI) and LCD translates the pipeline intent into Airflow DAGs.


## Weathering the Cloud Storms With Multi-Region Airflow Workflows
[URL](https://airflowsummit.org/sessions/2024/weathering-the-cloud-storms-with-multi-region-airflow-workflows/)  
Date: 2024-09-11 11:30:00
Summary:
Cloud availability zones and regions are not immune to outages. These zones regularly go down, and regions become unavailable due to natural disasters or human-caused incidents. Thus, if an availability zone or region goes down, so do your Airflow workflows and applications… unless your Airflow workflows function across multiple geographic locations.
This hands-on session introduces you to the design patterns of multi-region Airflow workflows in the cloud, which can tolerate zone and region-level incidents. We will start with a traditional single-region configuration and then switch to a multi-region setting. By the end, we’ll have a working prototype of a multi-region Airflow pipeline that recovers from region-level outages within a few seconds, with no data loss or disruption to the application layer.


## Airflow and Control-M: Where Data Pipelines Meet Business Applications in Production
[URL](https://airflowsummit.org/sessions/2024/airflow-and-control-m-where-data-pipelines-meet-business-applications-in-production/)  
Date: 2024-09-11 12:00:00
Summary:
This talk is presented by BMC Software With Airflow’s mainstream acceptance in the enterprise, the operational challenges of running with applications in production have emerged.
At last year’s Airflow Summit in Toronto, three providers of Apache Airflow met to discuss “The Future of Airflow: What Users Want”. Among the user requirements in the session were:
An improved security model allowing “Alice” and “Bob” to run their single DAGs without each requiring a separate Airflow cluster, while still adhering to their organization’s compliance requirements.


## Investigating the Many Loops of the Airflow Scheduler
[URL](https://airflowsummit.org/sessions/2024/investigating-the-many-loops-of-the-airflow-scheduler/)  
Date: 2024-09-11 12:30:00
Summary:
The scheduler is unarguably the most important component of an Airflow cluster. It is also the most complex and misunderstood by practitioners and administrators alike.
In this talk, we will follow the path that a task instance takes to progress from creation to execution, and discuss the various configuration settings allowing users to tune the scheduler and executor to suit their workload patterns. Finally, we will dive deep into critical sections of the Airflow codebase and explore opportunities for optimization.


## Product Management perspective on Data Observability with Databand
[URL](https://airflowsummit.org/sessions/2024/databand/)  
Date: 2024-09-11 12:30:00
Summary:
In this session Steve Sawyer will discuss a case study for how IBM Data Observability with Databand, collects metadata to build historical baselines, detect anomalies and triage alerts to remediate data quality issues for you data pipelines and warehouses. Additionally, he will provide a Product perspective on the technologies IBM is building to meet the data observability needs across the enterprise, and how it relates to our investments in AI and Data Fabric.


## Stress-Free Airflow development: From Dev to Prod at Stripe
[URL](https://airflowsummit.org/sessions/2024/stress-free-airflow-development-from-dev-to-prod-at-stripe/)  
Date: 2024-09-11 12:30:00
Summary:
At Stripe, compliance with regulations is of utmost importance, and ensuring the integrity of production data is crucial. To address this challenge, Stripe developed a powerful system called User Scope Mode (USM), which allows users to safely and efficiently test new or existing Airflow pipelines without the risk of corrupting production data.
USM takes care of automatically overwriting the necessary configurations for Airflow pipelines, enabling users to test their production-ready pipelines locally with ease. This approach empowers Stripe’s teams to iterate and refine their workflows without the burden of manual setup or the fear of disrupting live operations.


## Why Do Airflow Tasks Fail? An Analysis through Machine Learning Techniques
[URL](https://airflowsummit.org/sessions/2024/why-do-airflow-tasks-fail-an-analysis-through-machine-learning-techniques/)  
Date: 2024-09-11 12:30:00
Summary:
There are 3 certainties in life: death, taxes, and data pipelines failing. Pipelines may fail for a number of reasons: you may run out of memory, your credentials may expire, an upstream data source may not be reliable, etc. But there are patterns we can learn from!
Join us as we walk through an analysis we’ve done on a massive dataset of Airflow failure logs. We’ll show how we used natural language processing and dimensionality reduction methods to explore the latent space of Airflow task failures in order to cluster, visualize, and understand failures.


## Scalable Development of Event Driven Airflow DAGs
[URL](https://airflowsummit.org/sessions/2024/scalable-development-of-event-driven-airflow-dags/)  
Date: 2024-09-11 14:30:00
Summary:
This usecase shows how we deal with data of different varieties from different sources. Each source sends data in different layout, timings, structures, location patterns sizes. The goal is to process the files within SLA and send them out. This a complex multi step processing pipeline that involves multiple spark jobs, api based integrations with microservices, resolving unique ids, deduplication and filtering. Note that this is an event driven system, but not a streaming data system. The files are of gigabyte scale, and each day the data being processed is of terabyte scale.


## The Essentials of Custom Executor Development
[URL](https://airflowsummit.org/sessions/2024/the-essentials-of-custom-executor-development/)  
Date: 2024-09-11 14:30:00
Summary:
Since version 2.7 and the advent of AIP-51, Airflow has started to fully support the creation of custom executors. Before we dive into the components of an executor and how they work, we will briefly discuss the Executor Decoupling initiative which allowed this new feature. Once we understand the parts required, we will explore the process of crafting our own executors, using real-world examples, and demonstrations of executors developed within the Amazon Provider Package as a guide. By demystifying the process of executor creation and emphasizing the opportunities for contribution, we aim to empower Airflow users and providers to harness the full potential of custom executors, enriching the Airflow ecosystem as a whole!


## Workshop: DAG writing best practices
[URL](https://airflowsummit.org/sessions/2024/workshop-dags/)  
Date: 2024-09-11 14:30:00
Summary:
Learn how to design your DAGs for easy readability and maintenance, how to make your DAGs dynamic in an efficient and scalable way, and how to avoid common pitfalls that can cause performance issues.

## A Game of Constant Learning & Adjustment: Orchestrating ML Pipelines at the Philadelphia Phillies
[URL](https://airflowsummit.org/sessions/2024/a-game-of-constant-learning-adjustment-orchestrating-ml-pipelines-at-the-philadelphia-phillies/)  
Date: 2024-09-11 15:05:00
Summary:
When developing Machine Learning (ML) models, the biggest challenges are often infrastructural. How do we deploy our model and expose an inference API? How can we retrain? Can we continuously evaluate performance and monitor model drift?
In this talk, we will present how we are tackling these problems at the Philadelphia Phillies by developing a suite of tools that enable our software engineering and analytics teams to train, test, evaluate, and deploy ML models - that can be entirely orchestrated in Airflow. This framework abstracts away the infrastructural complexities that productionizing ML Pipelines presents and allows our analysts to focus on developing robust baseball research for baseball operations stakeholders across player evaluation, acquisition, and development.


## Building in Resource Awareness and Event Dependency into Airflow
[URL](https://airflowsummit.org/sessions/2024/building-in-resource-awareness-and-event-dependency-into-airflow/)  
Date: 2024-09-11 15:05:00
Summary:
In this talk, we will explore how adding custom dependency checks into Airflow’s scheduling system can elevate Airflow’s performance.
We will specifically discuss how we added general upstream events dependency checking as well as how to make Airflow aware of used/available compute resources so that the system can better decide when and where to run a given task on Kubernetes infrastructure.
We’ll cover why the existing dependency checking in Airflow is not sufficient in our use case, and why adding custom code to Airflow is needed. We’ll cover the pros and cons with this approach.


## Hybrid Executors: Have Your Cake and Eat it Too
[URL](https://airflowsummit.org/sessions/2024/hybrid-executors-have-your-cake-and-eat-it-too/)  
Date: 2024-09-11 15:05:00
Summary:
Executors are a core concept in Apache Airflow and they are an essential piece to the execution of DAGs. They continue to see investment and innovation including a new feature launching this year: Hybrid Execution.
This talk will give a brief overview of executors, how they work and what they are responsible for. Followed by a description of Hybrid Executors (AIP-61), a new feature to allow multiple executors to be used natively and seamlessly side by side within a single Airflow environment. We’ll deep dive into how this feature works, how users can make use of it, compare this new feature to what was available before, and finally a demo to see it in action. Don’t miss this chance to learn about the cutting edge capabilities of executors in Apache Airflow!


## Architecting Blockchain ETL Orchestration: Circle's Airflow Usecase
[URL](https://airflowsummit.org/sessions/2024/architecting-blockchain-etl-orchestration-circle-s-airflow-usecase/)  
Date: 2024-09-11 15:40:00
Summary:
This talk focuses on exploring the implementation of Apache Airflow for Blockchain ETL orchestration, indexing, and the adoption of GitOps at Circle. IT will cover CICD tips, architectural choices for managing Blockchain data at scale, engineering practices to enable data scientists and some learnings from production.


## Converting Legacy Schedulers to Airflow
[URL](https://airflowsummit.org/sessions/2024/converting-legacy-schedulers-to-airflow/)  
Date: 2024-09-11 15:40:00
Summary:
Having helped many customers to migrate thousands of workloads, we will discuss the process of migrations, and how we built an open-source framework to migrate legacy scheduler workflows via standard sets of patterns to Airflow Projects.
This framework is easily extended to encompass schedulers such as Automic, Autosys, Oozie, JAMS, SSIS and others, and has turned a difficult process requiring months or years to a simple one taking days or weeks.


## What If...? Running Airflow Tasks without the workers
[URL](https://airflowsummit.org/sessions/2024/what-if-running-airflow-tasks-without-the-workers/)  
Date: 2024-09-11 15:40:00
Summary:
Airflow executes all tasks on the workers, including deferrable operators that must run on the workers before deferring to the triggerer. However, running some tasks directly from the triggerer can be beneficial in certain situations. This presentation will explain how deferrable operators function and examine ways to modify the Airflow implementation to enable tasks to run directly from the triggerer.


## OpenLineage: From Operators to Hooks
[URL](https://airflowsummit.org/sessions/2024/openlineage-from-operators-to-hooks/)  
Date: 2024-09-11 16:30:00
Summary:
“More data lineage” has been second most popular feature request in Airflow Survey 2023. However, despite the integration of OpenLineage in Airflow 2.7 through AIP-53, the most popular Operator in Airflow - PythonOperator - isn’t covered by lineage support.
With addition of TaskFlow API, Airflow Datasets, Airflow ObjectStore, and many other small changes, writing DAGs without using other operators is easier than ever. And that’s why lineage collection in Airflow moves beyond covering specific Operators, to covering Hooks and Object Storage.


## Profiling Airflow tasks with Memray
[URL](https://airflowsummit.org/sessions/2024/profiling-airflow-tasks-with-memray/)  
Date: 2024-09-11 16:30:00
Summary:
Profiling Airflow tasks can be difficult, specially in remote environments. In this talk I will demonstrate how we can leverage the capabilities of Airflow’s plugin mechanism to selectively run Airflow tasks within the context of a profiler and with the help of operator links and custom views make the results available to the user.
The content of this talk can provide inspiration on how Airflow may in the future allow the gathering of custom task metrics and make those metrics easily accessible.


## Unlocking the Power of AI at Ford: A Behind-the-Scenes Look at Mach1ML and Airflow
[URL](https://airflowsummit.org/sessions/2024/unlocking-the-power-of-ai-at-ford-a-behind-the-scenes-look-at-mach1ml-and-airflow/)  
Date: 2024-09-11 16:30:00
Summary:
Ford Motor Company is undergoing a significant transformation, embracing AI and Machine Learning to power its smart mobility strategy, enhance customer experiences, and drive innovation in the automotive industry. Mach1ML, Ford’s multi-million dollar ML platform, plays a crucial role in this journey by empowering data scientists and engineers to efficiently build, deploy, and manage ML models at scale. This presentation will delve into how Mach1ML leverages Apache Airflow as its orchestration layer to tackle the challenges of complex ML workflows that include disparate systems, manual processes, security concerns, and deployment complexities. We will explore the benefits of using Airflow, such as increased efficiency, improved reliability, enhanced scalability, and faster time-to-value. Additionally, we will showcase how Mach1ML utilizes Airflow capabilities to generate reusable templates and streamline environment promotions to further empower Ford’s AI practitioners and accelerate the delivery of cutting-edge AI-powered solutions supporting the next generation of vehicles.


## Airflow and multi-cluster Slurm working together
[URL](https://airflowsummit.org/sessions/2024/airflow-and-multi-cluster-slurm-working-together/)  
Date: 2024-09-11 17:05:00
Summary:
Meteosim provides environmental services, mainly based on weather and air quality intelligence, and helps customers make operational and tactical decisions and understand their companies’ environmental impact. We introduced Airflow a couple of years ago to replace a huge Crontab file and we currently have around 7000 DAG Runs per day.
In this presentation we will introduce the hardest challenge we had to overcome: adapting Airflow to run on multiple Slurm-managed HPC clusters by using deferrable operators. Slurm is an open-source cluster manager, used especially in science-based companies or organizations and many supercomputers worldwide. By using Slurm our simulations run on bare metal nodes, eliminating overhead and speeding up the intensive calculations.


## How we Tuned our Airflow to Make 1.2 million DAG Runs - per day!
[URL](https://airflowsummit.org/sessions/2024/how-we-tuned-our-airflow-to-make-1-2-million-dag-runs-per-day/)  
Date: 2024-09-11 17:05:00
Summary:
As we deployed Airflow in our enterprise connected to various event sources to implement our data-driven pipelines we were faced with event storms a couple of times. As of such event storms happened often unplanned and with increased load waves we iteratively tuned the setup in multiple iterations. We were in panic and also needed to add some quick workarounds sometime.
Starting from a peak of 1000 triggers in a hour we were happy that workload just queued. But at a certain point we started tuning the setup. With about 10-20 iterations which we would like to share as best practice we started tuning standard parameters, increased resources, changed integration strategies as well and developed patches to core scheduler.


## Simplified user management in Airflow
[URL](https://airflowsummit.org/sessions/2024/simplified-user-management-in-airflow/)  
Date: 2024-09-11 17:05:00
Summary:
Before Airflow 2.9, user management was part of core Airflow, therefore modifying it or customizing it to fit user needs was not an easy process. Authentication and authorization managers (auth managers), is a new concept introduced in Airflow 2.9. It was introduced as extensible user management (AIP-56), allowing Airflow users to have a flexible way to integrate with organization’s identity services. Organizations want a single place to manage permissions and FAB (Flask App Builder) made it difficult to achieve. In this talk, after explaining the concept of auth managers and why we built this, we will show you how you can leverage the new auth manager interface to build an authorization service for Airflow based on your existing identity provider. We will see that auth managers can be leveraged to change considerably how users and their permissions are managed in an Airflow environment.


## Bronco: Managing Terraform at Scale with Airflow
[URL](https://airflowsummit.org/sessions/2024/bronco-managing-terraform-at-scale-with-airflow/)  
Date: 2024-09-11 17:40:00
Summary:
Airflow is not just purpose-built for data applications. It is a job scheduler on steroids. This is exactly what a cloud platform team needs: a configurable and scalable automation tool that can handle thousands of administrative tasks.
Come learn how one enterprise platform team used Airflow to support cloud infrastructure at unprecedented scale.


## How we Run 100 Airflow Environments and Millions of Tasks as a Part Time Job Using Kubernetes
[URL](https://airflowsummit.org/sessions/2024/how-we-run-100-airflow-environments-and-millions-of-tasks-as-a-part-time-job-using-kubernetes/)  
Date: 2024-09-11 17:40:00
Summary:
Balyasny Asset Management (BAM) is a diversified global investment firm founded in 2001 with over $20 billion in assets under management. We have more than 100 teams who run a variety of workloads that benefit from Orchestration and parallelization.
Platform Engineers working for companies with K8s ecosystems can use their Kubernetes knowledge and leverage their platform to run Airflow and troubleshoot problems successfully. BAM’s Kubernetes Platform provides production-ready Airflow environments that automatically get Logging, Metrics, Alerting, Scalability, Storage from a range of File Systems, Authentication, Dashboards, Secrets Management, and specialized compute including GPU, CPU Optimized, Memory Optimized and even Windows. If you can run thousands of Pods on your Kubernetes Cluster then you can run thousands of Tasks without needing to do anything! The intention of this talk is to cover:


## Mastering Advanced Dataset Scheduling in Apache Airflow
[URL](https://airflowsummit.org/sessions/2024/mastering-advanced-dataset-scheduling-in-apache-airflow/)  
Date: 2024-09-11 17:40:00
Summary:
Are you looking to harness the full potential of data-driven pipelines with Apache Airflow? This session will dive into the newly introduced conditional expressions for advanced dataset scheduling in Airflow - a feature highly requested by the Airflow community. Attendees will learn how to effectively use logical operators to create complex dependencies that trigger DAGs based on the dataset updates in real-world scenarios. We’ll also explore the innovative DatasetOrTimeSchedule, which combines time-based and dataset-triggered scheduling for unparalleled flexibility. Furthermore, attendees will discover the latest API endpoints that facilitate external updates and resets of dataset events, streamlining workflow management across different deployments.


## The road ahead: What’s coming in Airflow 3 and beyond?
[URL](https://airflowsummit.org/sessions/2024/the-road-ahead-whats-coming-in-airflow-3-and-beyond/)  
Date: 2024-09-12 09:00:00
Summary:
Apache Airflow has emerged as the defacto standard for data orchestration. Over the last couple of years, Airflow has also seen increasing adoption for ML and AI use cases. It has been almost four years since the release of Airflow 2 and as a community we have agreed that it’s time for a major foundational release in the form of Airflow 3.
This talk will introduce the vision behind Airflow 3, including the emerging technology trends in the industry and how Airflow will evolve in response. Specifically, this will include an overview of the architectural changes in Airflow to support emerging use cases and distributed data infrastructure models. This talk will also introduce the major features and the desired outcomes of the release. Airflow 3 will be a foundational release and therefore this talk will similarly introduce the new concepts being introduced as part of Airflow 3, which may be fully realized in follow-on 3.x releases.


## Airflow 3 - Roadmap Discussion
[URL](https://airflowsummit.org/sessions/2024/airflow-3-roadmap-discussion/)  
Date: 2024-09-12 09:25:00
Summary:
Join us in this panel with key members of the community behind the development of Apache Airflow where we will discuss the tentative scope for the next generation, i.e. Airflow 3.


## Airflow - Path to Industry Orchestration Standard
[URL](https://airflowsummit.org/sessions/2024/session-presented-by-google-cloud/)  
Date: 2024-09-12 10:30:00
Summary:
In the realm of data engineering, machine learning pipelines and using cloud and web services there is a huge demand for orchestration technologies.
Apache Airflow belongs to the most popular orchestration technologies or even is the most popular one.
In this presentation we are going to focus these aspects of Airflow that make it so popular and whether it became the orchestration industry standard.


## Connecting the Dots in Airflow: From User to Contributor
[URL](https://airflowsummit.org/sessions/2024/connecting-the-dots-in-airflow-from-user-to-contributor/)  
Date: 2024-09-12 10:30:00
Summary:
“Connecting the Dots in Airflow: From User to Contributor” explores the journey of transitioning from an Airflow user to an active project contributor. This talk will cover essential steps, resources, and best practices to effectively engage with the Airflow community and make meaningful contributions. Attendees will gain insights into the collaborative nature of open-source projects and how their involvement can drive both personal growth and project innovation.


## Scale and Security: How Autodesk Securely Develops and Tests PII Pipelines with Airflow
[URL](https://airflowsummit.org/sessions/2024/scale-and-security-how-autodesk-securely-develops-and-tests-pii-pipelines-with-airflow/)  
Date: 2024-09-12 10:30:00
Summary:
In today’s data-driven era, ensuring data reliability and enhancing our testing and development capabilities are paramount. Local unit testing has its merits but falls short when dealing with the volume of big data. One major challenge is running Spark jobs pre-deployment to ensure they produce expected results and handle production-level data volumes.
In this talk, we will discuss how Autodesk leveraged Astronomer to improve pipeline development. We’ll explore how it addresses challenges with sensitive and large data sets that cannot be transferred to local machines or non-production environments. Additionally, we’ll cover how this approach supports over 10 engineers working simultaneously on different feature branches within the same repo.


## Scaling Airflow for Data Productivity at Instacart
[URL](https://airflowsummit.org/sessions/2024/scaling-airflow-for-data-productivity-at-instacart/)  
Date: 2024-09-12 10:30:00
Summary:
In this talk, we’ll discuss how Instacart leverages Apache Airflow to orchestrate a vast network of data pipelines, powering both our core infrastructure and dbt deployments. As a data-driven company, Airflow plays a critical role in enabling us to execute large and intricate pipelines securely, compliantly, and at scale.
We’ll delve into the following key areas:
a. High-Throughput Cluster Management: We’ll explore how we manage and maintain our Airflow cluster, ensuring the efficient execution of over 2,000 DAGs across diverse use cases.


## Airflow, Spark, and LLMs: Turbocharging MLOps at ASAPP
[URL](https://airflowsummit.org/sessions/2024/airflow-spark-and-llms-turbocharging-mlops-at-asapp/)  
Date: 2024-09-12 11:00:00
Summary:
This talk will explore ASAPP’s use of Apache Airflow to streamline and optimize our machine learning operations (MLOps). Key highlights include:
Integrating with our custom Spark solution for achieving speedup, efficiency, and cost gains for generative AI transcription, summarization and intent categorization pipelines
Different design patterns of integrating with efficient LLM servers - like TGI/vllm/tensor-RT for Summarization pipelines with/without Spark.
An overview of batched LLM inference using Airflow as opposed to real time inference outside of it


## Empowering Airflow Users: A framework for performance testing and transparent resource optimization
[URL](https://airflowsummit.org/sessions/2024/empowering-airflow-users-a-framework-for-performance-testing-and-transparent-resource-optimization/)  
Date: 2024-09-12 11:00:00
Summary:
Apache Airflow is the backbone of countless data pipelines, but optimizing performance and resource utilization can be a challenge. This talk introduces a novel performance testing framework designed to measure, monitor, and improve the efficiency of Airflow deployments. I’ll delve into the framework’s modular architecture, showcasing how it can be tailored to various Airflow setups (Docker, Kubernetes, cloud providers). By measuring key metrics across schedulers, workers, triggers, and databases, this framework provides actionable insights to identify bottlenecks and compare performance across different versions or configurations.


## Unlocking the Power of Airflow Beyond Data Engineering at Cloudflare
[URL](https://airflowsummit.org/sessions/2024/unlocking-the-power-of-airflow-beyond-data-engineering-at-cloudflare/)  
Date: 2024-09-12 11:00:00
Summary:
While Airflow is widely known for orchestrating and managing workflows, particularly in the context of data engineering, data science, ML (Machine Learning), and ETL (Extract, Transform, Load) processes, its flexibility and extensibility make it a highly versatile tool suitable for a variety of use cases beyond these domains. In fact, Cloudflare has publicly shared in the past an example on how Airflow was leveraged to build a system that automates datacenter expansions.


## DAGify - Enterprise Scheduler Migration Accelerator for Airflow
[URL](https://airflowsummit.org/sessions/2024/dagify-enterprise-scheduler-migration-accelerator-for-airflow/)  
Date: 2024-09-12 11:30:00
Summary:
DAGify is a highly extensible, template driven, enterprise scheduler migration accelerator that helps organizations speed up their migration to Apache Airflow. While DAGify does not claim to migrate 100% of existing scheduler functionality it aims to heavily reduce the manual effort it takes for developers to convert their enterprise scheduler formats into Python Native Airflow DAGs.
DAGify is an open source tool under Apache 2.0 license and available on Github (https://github.com/GoogleCloudPlatform/dagify).


## Evolution of Airflow at Uber
[URL](https://airflowsummit.org/sessions/2024/evolution-of-airflow-at-uber/)  
Date: 2024-09-12 11:30:00
Summary:
Up until a few years ago, teams at Uber used multiple data workflow systems, with some based on open source projects such as Apache Oozie, Apache Airflow, and Jenkins while others were custom built solutions written in Python and Clojure.
Every user who needed to move data around had to learn about and choose from these systems, depending on the specific task they needed to accomplish. Each system required additional maintenance and operational burdens to keep it running, troubleshoot issues, fix bugs, and educate users.


## Gen AI using Airflow 3: A vision for Airflow RAGs
[URL](https://airflowsummit.org/sessions/2024/gen-ai-using-airflow-3-a-vision-for-airflow-rags/)  
Date: 2024-09-12 11:30:00
Summary:
Gen AI has taken the computing world by storm. As Enterprises and Startups have started to experiment with LLM applications, it has become clear that providing the right context to these LLM applications is critical.
This process known as Retrieval augmented generation (RAG) relies on adding custom data to the large language model, so that the efficacy of the response can be improved. Processing custom data and integrating with Enterprise applications is a strength of Apache Airflow.


## How we use Airflow at Booking to Orchestrate Big Data Workflows
[URL](https://airflowsummit.org/sessions/2024/how-we-use-airflow-at-booking-to-orchestrate-big-data-workflows/)  
Date: 2024-09-12 11:30:00
Summary:
The talk will cover how we use Airflow at the heart of our Workflow Management Platform(WFM) at Booking.com, enabling our internal users to orchestrate big data workflows on Booking Data Exchange(BDX).
High level overview of the talk:
Adapting open source Airflow helm chart to spin up Airflow installation in Booking Kubernetes Service (BKS) Coming up with Workflow definition format (yaml) Conversion of workflow.yaml to workflow.py DAGs Usage of Deferrable operators to provide standard step templates to users Workspaces (collection of workflows), using it to ensure role based access to DAG permissions for users Using okta for authentication Alerting, monitoring, logging Plans to shift to Astronomer 

## Airflow Unleashed: Making Hundreds of Deployments A Day at Coinbase
[URL](https://airflowsummit.org/sessions/2024/airflow-unleashed-making-hundreds-of-deployments-a-day-at-coinbase/)  
Date: 2024-09-12 12:00:00
Summary:
At Coinbase, Airflow is the backbone of ELT, supported by a vibrant community of over 500 developers. This vast engagement results in a continuous stream of enhancements, with hundreds of commits tested and released daily. However, this scale of development presents its own set of challenges, especially in deployment velocity.
Traditional deployment methodologies proved inadequate, significantly impeding the productivity of our developers. Recognizing the critical need for a solution that matches our pace of innovation, we developed AirAgent: a bespoke, fully autonomous deployer designed specifically for Airflow. Capable of deploying updates hundreds of times a day on both staging and production environments, AirAgent has transformed our development lifecycle, enabling immediate iteration and drastically improving developer velocity.


## Lessons from the Ecosystem: What can Airflow Learn from Other Open-source Communities?
[URL](https://airflowsummit.org/sessions/2024/lessons-from-the-ecosystem-what-can-airflow-learn-from-other-open-source-communities/)  
Date: 2024-09-12 12:00:00
Summary:
The Apache Airflow community is so large and active that it’s tempting to take the view that “if it ain’t broke don’t fix it.”
In a community as in a codebase, however, improvement and attention are essential to sustaining growth. And bugs are just as inevitable in community management as they are in software development. If only the fixes were, too!
Airflow is large and growing because users love Airflow and our community. But what steps could be taken to enhance the typical user’s and developer’s experience of the community?


## Adaptive Memory Scaling for Robust Airflow Pipelines
[URL](https://airflowsummit.org/sessions/2024/adaptive-memory-scaling-for-robust-airflow-pipelines/)  
Date: 2024-09-12 12:30:00
Summary:
At Vibrant Planet, we’re on a mission to make the world’s communities and ecosystems more resilient in the face of climate change. Our cloud-based platform is designed for collaborative scenario planning to tackle wildfires, climate threats, and ecosystem restoration on a massive scale.
In this talk we will dive into how we are using Airflow. Particularly we will focus on how we’re making Airflow pipelines smarter and more resilient, especially when dealing with the task of processing large satellite imagery and other geospatial data.


## Building Reliable Data Products
[URL](https://airflowsummit.org/sessions/2024/reliable-data-products/)  
Date: 2024-09-12 12:30:00
Summary:
Data engineers have shifted from delivering data for internal analytics applications to customer-facing data products. And with that shift comes a whole new level of operational rigor necessary to instill trust and confidence in the data. How do you hold data pipelines to the same standards as traditional software applications? Can you apply principles learned from the field of SRE to the world of data?
In this talk, we’ll explore how we’ve seen this evolve in Astronomer’s customer base and highlight best practices learned from the most critical data product applications we’ve seen. We’ll hear from Astronomer’s own data team as they went through the transformation from analytics to data products. And we’ll showcase a new product we’re building to help data teams around the world solve exactly this problem!


## Seeing Clearly with Airflow: The Shift to Data-Aware Orchestration
[URL](https://airflowsummit.org/sessions/2024/seeing-clearly-with-airflow-the-shift-to-data-aware-orchestration/)  
Date: 2024-09-12 12:30:00
Summary:
As Apache Airflow evolves, a key shift is emerging: the move from task-centric to data-aware orchestration. Traditionally, Airflow has focused on managing tasks efficiently, with limited visibility into the data those tasks manipulate. However, the rise of data-centric workflows demands a new approach—one that puts data at the forefront.
This talk will explore how embedding deeper data insights into Airflow can align with modern users’ needs, reducing complexity and enhancing workflow efficiency. We’ll discuss how this evolution can transform Airflow into a more intuitive and powerful tool, better suited to today’s data-driven environments.


## The Silent Symphony: Keeping Airflow's CI/CD and Dev Tools in Tune
[URL](https://airflowsummit.org/sessions/2024/the-silent-symphony-keeping-airflow-s-ci-cd-and-dev-tools-in-tune/)  
Date: 2024-09-12 12:30:00
Summary:
Apache Airflow relies on a silent symphony behind the scenes: its CI/CD (Continuous Integration/Continuous Delivery) and development tooling. This presentation explores the critical role these tools play in keeping Airflow efficient and innovative. We’ll delve into how robust CI/CD ensures bug fixes and improvements are seamlessly integrated, while well-maintained development tools empower developers to contribute effectively.
Airflow’s power comes from a well-oiled machine – its CI/CD and development tools. This presentation dives into the world of these often-overlooked heroes. We’ll explore how seamless CI/CD pipelines catch and fix issues early, while robust development tools empower efficient coding and collaboration. Discover how you can use and contribute to a thriving Airflow ecosystem by ensuring these crucial tools stay in top shape.


## AIP-63: DAG Versioning - Where are we?
[URL](https://airflowsummit.org/sessions/2024/aip-63-dag-versioning-where-are-we/)  
Date: 2024-09-12 14:00:00
Summary:
Join us as we check in on the current status of AIP-63: DAG Versioning. This session will explore the motivations behind AIP-63, the challenges faced by Airflow users in understanding and managing DAG history, and how it aims to address them. From tracking TaskInstance history to improving DAG representation in the UI, we’ll examine what we’ve already done and what’s next. We’ll also touch upon the potential future steps outlined in AIP-66 regarding the execution of specific DAG versions.


## Empowering Business Analysts with DAG Authoring IDE Running 8000 Workflows
[URL](https://airflowsummit.org/sessions/2024/empowering-business-analysts-with-dag-authoring-ide-running-8000-workflows/)  
Date: 2024-09-12 14:00:00
Summary:
At Wix more often than not business analysts build workflows themselves to avoid data engineers being a bottleneck. But how do you enable them to create SQL ETLs starting when dependencies are ready and sending emails or refreshing Tableau reports when the work is done? One simple answer may be to use Airflow. The problem is every BA cannot be expected to know Python and Git so well that they will create thousands of DAGs easily.


## How the Airflow Community Productionizes Generative AI
[URL](https://airflowsummit.org/sessions/2024/how-the-airflow-community-productionizes-generative-ai/)  
Date: 2024-09-12 14:00:00
Summary:
Every data team out there is being asked from their business stakeholders about Generative AI. Taking LLM centric workloads to production is not a trivial task. At the foundational level, there are a set of challenges around data delivery, data quality, and data ingestion that mirror traditional data engineering problems. Once you’re past those, there’s a set of challenges related to the underlying use case you’re trying to solve. Thankfully, because of how Airflow was already being used at these companies for data engineering and MLOps use cases, it has become the defacto orchestration layer behind many GenAI use cases for startups and Fortune 500s.


## Airflow-as-an-Engine: Lessons from Open-Source Applications Built On Top of Airflow
[URL](https://airflowsummit.org/sessions/2024/airflow-as-an-engine-lessons-from-open-source-applications-built-on-top-of-airflow/)  
Date: 2024-09-12 14:35:00
Summary:
Airflow is often used for running data pipelines, which themselves connect with other services through the provider system. However, it is also increasingly used as an engine under-the-hood for other projects building on top of the DAG primitive. For example, Cosmos is a framework for automatically transforming dbt DAGs into Airflow DAGs, so that users can supplement the developer experience of dbt with the power of Airflow.
This session dives into how a select group of these frameworks (Cosmos, Meltano, Chronon) use Airflow as an engine for orchestrating complex workflows their systems depend on. In particular, we will discuss ways that we’ve increased Airflow performance to meet application-specific demands (high-task-count Cosmos DAGs, streaming jobs in Chronon), new Airflow features that will evolve how these frameworks use Airflow under the hood (DAG versioning, dataset integrations), and paths we see these projects taking over the next few years as Airflow grows. Airflow is not just a DAG platform, it’s an application platform!


## Using Airflow operational data to optimize Cloud services
[URL](https://airflowsummit.org/sessions/2024/using-airflow-operational-data-to-optimize-cloud-services/)  
Date: 2024-09-12 14:35:00
Summary:
Cost management is a continuous challenge for our data teams at Astronomer. Understanding the expenses associated with running our workflows is not always straightforward, and identifying which process ran a query causing unexpected usage on a given day can be time-consuming.
In this talk, we will showcase an Airflow Plugin and specific DAGs developed and used internally at Astronomer to track and optimize the costs of running DAGs. Our internal tool monitors Snowflake query costs, provides insights, and sends alerts for abnormal usage. With it, Astronomer identified and refactored its most costly DAGs, resulting in an almost 25% reduction in Snowflake spending.


## Customizing LLMs: Leveraging Technology to tailor GenAI using Airflow
[URL](https://airflowsummit.org/sessions/2024/customizing-llms-leveraging-technology-to-tailor-genai-using-airflow/)  
Date: 2024-09-12 15:10:00
Summary:
Laurel provides an AI-driven timekeeping solution tailored for accounting and legal firms, automating timesheet creation by capturing digital work activities. This session highlights two notable AI projects:
UTBMS Code Prediction: Leveraging small language models, this system builds new embeddings to predict work codes for legal bills with high accuracy. More details are available in our case study: https://www.laurel.ai/resources-post/enhancing-legal-and-accounting-workflows-with-ai-insights-into-work-code-prediction.
Bill Creation and Narrative Generation: Utilizing Retrieval-Augmented Generation (RAG), this approach transforms users’ digital activities into fully billable entries.


## Hello Quality: Building CIs to run Providers Packages System Tests
[URL](https://airflowsummit.org/sessions/2024/hello-quality-building-cis-to-run-providers-packages-system-tests/)  
Date: 2024-09-12 15:10:00
Summary:
Airflow operators are a core feature of Apache Airflow and it’s extremely important that we maintain high quality of operators, prevent regressions and on the other hand we help developers with automated tests results to double check if introduced changes don’t cause regressions or backward incompatible changes and we provide Airflow release managers with information whether a given version of a provider should be released or not yet.
Recently a new approach to assuring production quality was implemented for AWS, Google and Astronomer-provided operators - standalone Continuous Integration processes were configured for them and test results dashboards show the results of the last test runs. What has been working well for these operator providers might be a pattern to follow for others - during this presentation, AWS, Google and Astronomer engineers are going to share the information about the internals of Test Dashboards implemented for AWS, Google and Astronomer-provided operators. This approach might be a a ‘blueprint’ to follow for other providers.


## Using the power of Apache Airflow and Ray for Scalable AI deployments
[URL](https://airflowsummit.org/sessions/2024/using-the-power-of-apache-airflow-and-ray-for-scalable-ai-deployments/)  
Date: 2024-09-12 15:10:00
Summary:
Many organizations struggle to create a well-orchestrated AI infrastructure, using separate and disconnected platforms for data processing, model training, and inference, which slows down development and increases costs. There’s a clear need for a unified system that can handle all aspects of AI development and deployment, regardless of the size of data or models.
Join our breakout session to see how our comprehensive solution simplifies the development and deployment of large language models in production. Learn how to streamline your AI operations by implementing an end-to-end ML lifecycle on your custom data, including - automated LLM fine-tuning, LLM evaluation & LLM serving and LoRA deployments


## Exploring DAG Design Patterns in Apache Airflow
[URL](https://airflowsummit.org/sessions/2024/exploring-dag-design-patterns-in-apache-airflow/)  
Date: 2024-09-12 15:45:00
Summary:
This talk delves into advanced Directed Acyclic Graph (DAG) design patterns that are pivotal for optimizing data pipeline management and boosting efficiency. We’ll cover dynamic DAG generation, which allows for flexible, scalable workflow creation based on real-time data and configurations. Learn about task grouping and SubDAGs to enhance readability and maintainability of complex workflows. We’ll also explore parameterized DAGs for injecting runtime parameters into tasks, enabling versatile and adaptable pipeline configurations. Additionally, the session will address branching and conditional execution to manage workflow paths dynamically based on data conditions or external triggers. Lastly, understand how to leverage parallelism and concurrency to maximize resource utilization and reduce execution times. This session is designed for intermediate to advanced users who are familiar with the basics of Airflow and looking to deepen their understanding of its more sophisticated capabilities.


## Refactoring DAGs: From Duplication to Delightful Efficiency with a Centralized Library
[URL](https://airflowsummit.org/sessions/2024/refactoring-dags/)  
Date: 2024-09-12 15:45:00
Summary:
Feeling trapped in a maze of duplicate Airflow DAG code? We were too! That’s why we embarked on a journey to build a centralized library, eliminating redundancy and unlocking delightful efficiency.
Join us as we share:
The struggles of managing repetitive code across DAGs Our approach to a centralized library, revealing design and implementation strategies The amazing results: reduced development time, clean code, effortless maintenance, and a framework that creates efficient and self-documenting DAGs Let’s break free from complexity and duplication, and build a brighter Airflow future together!


## Introducing Apache Airflow® 3 – The Next Evolution in Orchestration
[URL](https://airflowsummit.org/sessions/2025/introducing-apache-airflow-3/)  
Date: 2025-10-07 09:00:00
Summary:
Apache Airflow® 3 is here, bringing major improvements to data orchestration. In this keynote, core Airflow contributors will walk through key enhancements that boost flexibility, efficiency, and user experience.
Vikram Koka will kick things off with an overview of Airflow 3, followed by deep dives into DAG versioning (Jed Cunningham), enhanced backfilling (Daniel Standish), and a modernized UI (Brent Bovenzi & Pierre Jeambrun).
Next, Ash Berlin-Taylor, Kaxil Naik, and Amogh Desai will introduce the Task Execution Interface and Task SDK, enabling tasks in any environment and language. Jens Scheffler will showcase the Edge Executor, while Constance Martineau, Tzu-ping Chung and Vincent Beck will demo event-driven scheduling and data assets. Finally, Buğra Öztürk will unveil CLI enhancements for automation and debugging.


## Benchmarking the Performance of Dynamically Generated DAGs
[URL](https://airflowsummit.org/sessions/2025/benchmarking-the-performance-of-dynamically-generated-dags/)  
Date: 2025-10-07 11:30:00
Summary:
In this talk, we’ll share our approach to benchmarking dynamically generated DAGs with Astronomer Cosmos. By the end of this session, you will have practical benchmarks and strategies for making informed decisions about evaluating the performance of DAGs in Airflow.

## Linkedin's journey on scaling Airflow
[URL](https://airflowsummit.org/sessions/2025/linkedin-s-journey-on-scaling-airflow/)  
Date: 2025-10-07 11:30:00
Summary:
Following up from our session last year, we will take a deeper dive into the challenges, solutions, and engineering innovations that helped us scale Airflow to support thousands of concurrent tasks while maintaining usability and reliability.

## Security made us do it: Airflow’s new Task Execution Architecture
[URL](https://airflowsummit.org/sessions/2025/security-made-us-do-it-airflows-new-task-execution-architecture/)  
Date: 2025-10-07 11:30:00
Summary:
Join us to explore how AIP-72 transforms Airflow task execution, paving the way for a more secure, flexible, and futuristic task orchestration!

## Why AWS chose Apache Airflow to power workflows for the next generation of Amazon SageMaker
[URL](https://airflowsummit.org/sessions/2025/why-aws-chose-apache-airflow-to-power-workflows-for-the-next-generation-of-amazon-sagemaker/)  
Date: 2025-10-07 11:30:00
Summary:
This talk will go into details around the motivations for choosing Airflow for Amazon SageMaker, the main challenges we faced, the key role that open-source plays, how we’re leveraging GenAI to make that open source development experience better, and the goals for the future of Airflow in SageMaker Unified Studio.

## Airflow That Remembers: The Dag Versioning Era is here!
[URL](https://airflowsummit.org/sessions/2025/airflow-that-remembers-the-dag-versioning-era-is-here/)  
Date: 2025-10-07 12:15:00
Summary:
DAG versioning is one of the most important features introduced in Airflow 3. In this talk we will cover how DAG code is versioned, how to visualize changes over time in the UI, choosing versions to run a DAG and how to handle dynamic DAGs.

## Enhancing DAG Management with DMS: A Scalable Solution for Airflow
[URL](https://airflowsummit.org/sessions/2025/enhancing-dag-management-with-dms-a-scalable-solution-for-airflow/)  
Date: 2025-10-07 12:15:00
Summary:
We introduce the DAG Management Service (DMS), developed to address critical challenges in managing Airflow clusters. We will also discuss how DMS streamlines Airflow version upgrades. Upgrading from an old Airflow version to the latest requires sequential updates and code modifications for over 10,000 DAGs. DMS proposes an efficient upgrade method, reducing dependency on users.

## Lessons learned for scaling up Airflow 3 in Public Cloud
[URL](https://airflowsummit.org/sessions/2025/lessons-learned-for-scaling-up-airflow-3-in-public-cloud/)  
Date: 2025-10-07 12:15:00
Summary:
This talk shares performance results for various Airflow 3 configurations and provides potential Airflow 3 adopters good understanding of its performance.

## Why Datadog Chose Airflow 3: Multi-Tenancy, Observability, and the Future of Event-Driven Workflows
[URL](https://airflowsummit.org/sessions/2025/event-driven-partition-aware-modern-orchestration-with-airflow-at-datadog/)  
Date: 2025-10-07 12:15:00
Summary:
Datadog is a world-class data platform ingesting more than a 100 trillion events a day, providing real-time insights.
Before Airflow’s prominence, we built batch processing on Luigi, Spotify’s open-source orchestrator. As Airflow gained wide adoption, we evaluated adopting the major improvements of release 2.0, but opted for building our own orchestrator instead to realize our dataset-centric, event-driven vision.
Meanwhile, the 3.0 release aligned Airflow with the same vision we pursued internally, as a modern asset-driven orchestrator. It showed how futile it is to build our own compared to the momentum of the community. We evaluated several orchestrators and decided to join forces with the Airflow project.


## Airflow & Your Automation CoE: Streamlining Integration for Enterprise-Wide Governance and Value
[URL](https://airflowsummit.org/sessions/2025/airflow-and-your-automation-coe/)  
Date: 2025-10-07 14:00:00
Summary:
This session explores common challenges faced when bringing specialized tools like Airflow into a broader Automation Center of Excellence framework.

## Orchestrating Data Quality - Quality Data Brought To You By Airflow
[URL](https://airflowsummit.org/sessions/2025/orchestrating-data-quality-quality-data-brought-to-you-by-airflow/)  
Date: 2025-10-07 14:00:00
Summary:
We’ll explore how the Astronomer data team leverages Airflow to uphold data quality across complex pipelines; minimizing firefighting and maximizing confidence in reported metrics.

## Orchestrating Travel Insights: Priceline's MLOps with Airflow
[URL](https://airflowsummit.org/sessions/2025/orchestrating-travel-insights-priceline/)  
Date: 2025-10-07 14:00:00
Summary:
This session dives into how Apache Airflow serves as a unifying force in MLOps. We will start by analyzing the broader MLOps trends and how Airflow is evolving to meet these challenges and then we will share a case study of the MLOps transformation at Priceline.

## Unlocking Event-Driven Scheduling in Airflow 3: A New Era of Reactive Data Pipelines
[URL](https://airflowsummit.org/sessions/2025/unlocking-event-driven-scheduling-in-airflow-3-a-new-era-of-reactive-data-pipelines/)  
Date: 2025-10-07 14:00:00
Summary:
Airflow 3 introduces a major evolution in orchestration: native support for external event-driven scheduling. In this talk, I’ll share the journey behind AIP-82—why we needed it, how we built it, and what it unlocks.

## Automating Business Intelligence with Airflow: A Practical Guide
[URL](https://airflowsummit.org/sessions/2025/automating-business-intelligence-with-airflow-a-practical-guide/)  
Date: 2025-10-07 14:30:00
Summary:
This session offers a practical guide to automating business intelligence processes using Apache Airflow. We’ll walk through real-world examples of automating data extraction, transformation, dashboard refreshes, and report distribution.

## Beyond Logs: Unlocking Airflow 3.0 Observability with OpenTelemetry Traces
[URL](https://airflowsummit.org/sessions/2025/beyond-logs-unlocking-airflow-3-0-observability-with-opentelemetry-traces/)  
Date: 2025-10-07 14:30:00
Summary:
his presentation will explain the basics: what OpenTelemetry is and how it works – perfect for someone with no prior familiarity with tracing or with the use of OpenTelemetry. It will demonstrate how Airflow users can leverage the new tracing support to achieve deeper observability into DAG runs.

## DAGLint: Elevating Airflow DAG Quality Through Automated Linting
[URL](https://airflowsummit.org/sessions/2025/daglint-elevating-airflow-dag-quality-through-automated-linting/)  
Date: 2025-10-07 14:30:00
Summary:
We will share why we decided to build a custom, internally developed linting tool for Airflow DAGs, how we built it and how we use it to elevate our code quality.

## Orchestrating Databricks with Airflow: Unlocking the Power of MVs, Streaming Tables, and AI
[URL](https://airflowsummit.org/sessions/2025/orchestrating-databricks-with-airflow/)  
Date: 2025-10-07 14:30:00
Summary:
We’ll showcase how Airflow can automate and optimize Databricks workflows, reducing costs and improving performance for large-scale data processing.

## Breaking News with Data Pipelines: How Airflow and AI Power Investigative Journalism
[URL](https://airflowsummit.org/sessions/2025/breaking-news-with-data-pipelines/)  
Date: 2025-10-07 15:00:00
Summary:
We will share how we use Airflow at the Financial Times to process and analyze text, documents, and other difficult-to-structure data sources combining AI, machine learning, and advanced computational techniques to extract meaningful entities, relationships, and patterns.

## Deadline Alerts in Airflow 3.1
[URL](https://airflowsummit.org/sessions/2025/deadline-alerts-in-airflow-3-1/)  
Date: 2025-10-07 15:00:00
Summary:
Come learn about the all-new Deadline Alerts feature in Airflow 3.1 which replaces SLA. We will discuss how Deadline Alerts work and how they improve on the retired SLA feature.

## Orchestrator of Orchestrators: Uniting Airflow Pipelines with Business Applications in Production
[URL](https://airflowsummit.org/sessions/2025/orchestrator-of-orchestrators/)  
Date: 2025-10-07 15:00:00
Summary:
In this demo-driven session we will connect Airflow with Control-M from BMC and showcase how Airflow can participate in end-to-end workflows that span not just data platforms but also transactional business applications.

## Seamless Airflow Upgrades: Migrating from 2.x to 3
[URL](https://airflowsummit.org/sessions/2025/seamless-airflow-upgrades/)  
Date: 2025-10-07 15:00:00
Summary:
Join this session for live demos and practical examples that will empower you to confidently upgrade from Airflow 2, minimise downtime, and achieve optimal performance in Airflow 3.

## Airflow Without Borders: A Journey into Internationalization (i18n)
[URL](https://airflowsummit.org/sessions/2025/airflow-without-borders/)  
Date: 2025-10-07 15:45:00
Summary:
This talk will highlight the UI changes made to support different languages, including locale-aware adjustments. We’ll discuss how translations are contributed and managed — including the use of LLMs to accelerate the process — and why human review remains an essential part of it. We’ll present the i18n policy designed to ensure long-term maintainability, along with the tooling developed to support it.

## Designing Scalable Retrieval-Augmented Generation (RAG) Pipelines at SAP with Apache Airflow
[URL](https://airflowsummit.org/sessions/2025/designing-scalable-retrieval-augmented-generation-rag-pipelines-at-sap-with-apache-airflow/)  
Date: 2025-10-07 15:45:00
Summary:
Join our session to discover how SAP Business AI developed a cutting-edge system based on Airflow that effectively grounds Large Language Models (LLMs) with rich SAP enterprise data, enabling us to transform RAG pipelines into enterprise-grade powerhouses.

## LLM-Powered Review Analysis: Optimising Data Engineering using Airflow
[URL](https://airflowsummit.org/sessions/2025/llm-powered-review-analysis/)  
Date: 2025-10-07 15:45:00
Summary:
A real-world journey of how a small team at Xena Intelligence built robust data pipelines for our enterprise customers using Airflow. Learn how we orchestrated a complex workflow to process millions of public reviews.

## Pittsburgh Goes With The Flow - Use Cases In Local Government
[URL](https://airflowsummit.org/sessions/2025/pittsburgh-goes-with-the-flow-use-cases-in-local-government/)  
Date: 2025-10-07 15:45:00
Summary:
Come and learn how the City of Pittsburgh uses Airflow for a variety of tasks, from employee-focused use cases to publication of public-facing information. Airflow acts as a funnel point and is an essential tool for Pittsburgh’s Data Services team.

## Airflow at Zoox: A journey to orchestrate heterogeneous workflows
[URL](https://airflowsummit.org/sessions/2025/airflow-at-zoox-a-journey-to-orchestrate-heterogeneous-workflows/)  
Date: 2025-10-07 16:15:00
Summary:
This presentation will detail how a small team at Zoox has managed to address the challenge of using Airflow for orchestrating heterogeneous workflows encompassing data, ML, and QA pipelines. We will share our strategies for simplifying the Airflow DAG creation process and enhancing user experience.

## EdgeExecutor / Edge Worker - The new option to run anywhere
[URL](https://airflowsummit.org/sessions/2025/edgeexecutor-edge-worker-the-new-option-to-run-anywhere/)  
Date: 2025-10-07 16:15:00
Summary:
Airflow 3 extends the deployment options to run your workload anywhere. You don’t need to bring your data to airflow but you can bring the execution where it needs to be.
You can connect any cloud and on-prem location together and generate a hybrid workflow from one central Airflow instance. Only a HTTP connection is needed.
We will present the use cases and concepts of the Edge deployment and how it is working also in a hybrid setup with Celery or other executors.


## ELT and Elections: Cloud-agnostic patterns for real-time analysis
[URL](https://airflowsummit.org/sessions/2025/elt-and-elections/)  
Date: 2025-10-07 16:15:00
Summary:
Discover how Apache Airflow powers scalable ELT pipelines, enabling seamless data ingestion, transformation, and machine learning-driven insights.

## Scaling Airflow at OpenAI
[URL](https://airflowsummit.org/sessions/2025/airflow-openai/)  
Date: 2025-10-07 16:15:00
Summary:
This talk shares how we scaled and hardened OpenAI’s Airflow deployment to orchestrate thousands of workflows on Kubernetes.

## Allegro's Airflow Journey: From On-Prem to Cloud Orchestration at Scale
[URL](https://airflowsummit.org/sessions/2025/allegro-s-airflow-journey-from-on-prem-to-cloud-orchestration-at-scale/)  
Date: 2025-10-07 16:45:00
Summary:
his session will detail Allegro’s, a leading e-commerce company in Poland, journey with Apache Airflow. It will chart our evolution from a custom, on-premises Airflow-as-a-Service solution through a significant expansion to over 300 Cloud Composer instances in Google Cloud, culminating in Airflow becoming the core of our data processing.

## Implementing Airflow Governance with Cluster Policies
[URL](https://airflowsummit.org/sessions/2025/implementing-airflow-governance/)  
Date: 2025-10-07 16:45:00
Summary:
In this talk, Karen will take a more scenario-based approach, and talk about how cluster policies can be used to do things like prevent a Data Engineer from inserting/deleting data from a production database, or make sure that Tasks are always assigned to the right sized worker.

## From DAGs to Insights: Business-Driven Airflow Use Cases
[URL](https://airflowsummit.org/sessions/2025/from-dags-to-insights/)  
Date: 2025-10-08 09:00:00
Summary:
This session dives into use cases from GitHub where key business decisions are driven, at the root, with the help of Airflow.

## New Tools, Same Craft: The Developer's Toolbox in 2025
[URL](https://airflowsummit.org/sessions/2025/new-tools-same-craft/)  
Date: 2025-10-08 09:30:00
Summary:
In this keynote  I’ll share observations about how even though our development workflow and toolchain of is changing dramatically, the core skills of a good software engineer like problem-solving, collaborative debugging and systems thinking, remain as crucial as ever.

## Airflow 3 - An Open Heart Surgery
[URL](https://airflowsummit.org/sessions/2025/airflow-3-an-open-heart-surgery/)  
Date: 2025-10-08 10:30:00
Summary:
Curious how code truly flows inside Airflow? Join me for a unique visualisation journey into Airflow’s inner workings (first of its kind) — code blocks and modules called when certain operations are running.
A walkthrough that unveils task execution, observability, and debugging like never before. Scaling of Airflow in action, showing performance comparison b/w Airflow 3 vs 2. This session will demystify Airflow’s architecture, showcasing real-time task flows and the heartbeat of pipelines in action.


## Dynamic Data Pipelines with DBT and Airflow
[URL](https://airflowsummit.org/sessions/2025/dynamic-data-pipelines-with-dbt-and-airflow/)  
Date: 2025-10-08 10:30:00
Summary:
This session showcases Okta’s innovative approach to data pipeline orchestration with dbt and Airflow. How we’ve implemented dynamically generated airflow dags workflows based on dbt’s dependency graph. This allows us to enforce strict data quality standards by automatically executing downstream model tests before upstream model deployments, effectively preventing error cascades. The entire CI/CD pipeline, from dbt model changes to production DAG deployment, is fully automated. The result? Accelerated development cycles, reduced operational overhead, and bulletproof data reliability


## Get started with Airflow 3.0
[URL](https://airflowsummit.org/sessions/2025/get-started-with-airflow-3-0/)  
Date: 2025-10-08 10:30:00
Summary:
Get hands-on experience with the new release and learn how to leverage new features like DAG versioning, backfills, data assets, and a new react-based UI.

## Orchestrating Apache Airflow ML Workflows at Scale with SageMaker Unified Studio
[URL](https://airflowsummit.org/sessions/2025/orchestrating-apache-airflow-ml-workflows-at-sale-with-sagemaker-unified-studio/)  
Date: 2025-10-08 10:30:00
Summary:
We’ll explore how to leverage Amazon SageMaker Unified Studio to build and deploy scalable Apache Airflow workflows that span the data and AI/ML lifecycle.

## Simplifying DAG creation with an AI-powered IDE for Airflow
[URL](https://airflowsummit.org/sessions/2025/simplifying-dag-creation-with-an-ai-powered-ide-for-airflow/)  
Date: 2025-10-08 10:30:00
Summary:
As the demand for data products grows, data engineering teams face mounting pressure to deliver more and even faster, often becoming bottlenecks. Astro IDE changes the game.
Astro IDE is an AI-powered code editor built for Apache Airflow. It helps data teams go from idea to production in minutes—generating production-ready DAGs, enabling in-browser testing, and integrating directly with Git.
In this session, see how Astro IDE accelerates DAG creation, debugging, and deployment so data engineering teams can deliver more, 10x faster.


## The Secret to Airflow's Evergreen Build: CI/CD magic
[URL](https://airflowsummit.org/sessions/2025/the-secret-to-airflow-s-evergreen-build-ci-cd-magic/)  
Date: 2025-10-08 10:30:00
Summary:
Have you ever wondered why Apache Airflow builds are asymptotically(*) green? That thrive for “perennial green build” is not magic, it’s the result of continuous, often unseen engineering effort within our CI/CD pipelines & dev environments. This dedication ensures that maintainers can work efficiently & contributors can onboard smoothly.
To tackle the ever growing contributor base, we have a CI/CD team run by volunteers putting in significant work in the foundational tooling. In this talk, we reveal some innovative solutions we have implemented like:


## Unleash Airflow's Potential with hands-on Performance Optimization
[URL](https://airflowsummit.org/sessions/2025/unleash-airflowss-potential-with-hands-on-performance-optimization/)  
Date: 2025-10-08 10:30:00
Summary:
This interactive workshop session empowers you to unlock the full potential of Apache Airflow through performance optimization techniques.

## Airflow as an AI Agent's toolkit: Going beyond MCPs and unlocking Airflow's 1000+ Integrations
[URL](https://airflowsummit.org/sessions/2025/airflow-as-an-ai-agents-toolkit-unlocking-1000-integrations-with-mcp/)  
Date: 2025-10-08 11:15:00
Summary:
What if your Airflow tasks could understand natural language AND adapt to schema changes automatically, while maintaining the deterministic, observable workflows we rely on? This talk introduces practical patterns for AI-native orchestration that preserve Airflow’s strengths while adding intelligence where it matters most.
Through a real-world example, we’ll demonstrate AI-powered tasks that detect schema drift across multi-cloud systems and perform context-aware data quality checks that go beyond simple validation—understanding business rules, detecting anomalies, and generating validation queries from prompts like “check data quality across regions.” All within static DAG structures you can test and debug normally.


## Airflow DAG Upgrade Agent: Using Google ADK, Gemini, and Vertex AI RAG Engine to Accelerate Upgrades
[URL](https://airflowsummit.org/sessions/2025/dag-upgrade-agent/)  
Date: 2025-10-08 11:15:00
Summary:
Join us to explore the DAG Upgrade Agent. Developed with Google Agent Development Kit and powered by Gemini, the DAG Upgrade Agent uses a rules-based framework to analyze DAG code, identify compatibility issues between core airflow and provider package versions, and generates precise upgrade recommendations and automated code conversions. Perfect for upcoming Airflow 3.0 migrations.


## Managed Workflow for Apache Airflow (MWAA): What's New?
[URL](https://airflowsummit.org/sessions/2025/mwaa-whats-new/)  
Date: 2025-10-08 11:15:00
Summary:
MWAA is an AWS-managed service that simplifies the deployment and maintenance of the open-source Apache Airflow data orchestration platform. MWAA has recently introduced several new features to enhance the experience for data engineering teams. Features such as Graceful Worker Replacement Strategy that enable seamless MWAA environment updates with zero downtime, IPv6 support, and in place minor Airflow Version Downgrade are some of the many new improvements MWAA has brought to their users in 2025. Last, but not the least, the release of Airflow 3.0 support brings the latest open-source features introducing a new web-server UI, better isolation and security for environments. These enhancements demonstrate Amazon’s continued investment in making Airflow more accessible and scalable for enterprises through the MWAA service.


## Operation Airlift: Uber's ongoing journey of migrating 200K pipelines to a single Airflow3 instance
[URL](https://airflowsummit.org/sessions/2025/operation-airlift-uber-s-ongoing-journey-of-migrating-200k-pipelines-to-a-single-airflow3-instance/)  
Date: 2025-10-08 11:15:00
Summary:
Yes, you read that right — 200,000 pipelines, nearly 1 million task executions per day, all powered by a single Airflow instance.
In this session, we’ll take you behind the scenes of one of the boldest orchestration projects ever attempted: how Uber’s data platform team is executing what might be the largest Apache Airflow migration in history — and doing it straight to Airflow 3.
From scaling challenges and architectural choices to lessons learned in high-throughput orchestration, this is a deep dive into the tech, the chaos, and the strategy behind making data fly at unprecedented scale.


## Agentic AI Automating Semantic Layer Updates with Airflow 3
[URL](https://airflowsummit.org/sessions/2025/agentic-ai-automating-semantic-layer-updates-with-airflow-3/)  
Date: 2025-10-08 12:00:00
Summary:
In today’s dynamic data environments, tables and schemas are constantly evolving and keeping semantic layers up to date has become a critical operational challenge. Manual updates don’t scale, and delays can quickly lead to broken dashboards, failed pipelines, and lost trust.
We’ll show how to harness Apache Airflow 3 and its new event-driven scheduling capabilities to automate the entire lifecycle: detecting table and schema changes in real time, parsing and interpreting those changes, and shifting left the updating of semantic models across dbt, Looker, or custom metadata layers. AI agents will add intelligence and automation that rationalize schema diffs, assess impact of changes, and propose targeted updates to semantic layers reducing manual work and minimizing the risk of errors.


## Boosting dbt-core workflows performance with Airflow’s Deferrable capabilities
[URL](https://airflowsummit.org/sessions/2025/boosting-dbt-core-workflows-performance-with-airflows-deferrable-capabilities/)  
Date: 2025-10-08 12:00:00
Summary:
Efficiently handling long-running workflows is crucial for scaling modern data pipelines. Apache Airflow’s deferrable operators help offload tasks during idle periods — freeing worker slots while tracking progress.
This session explores how Cosmos 1.9 (https://github.com/astronomer/astronomer-cosmos) integrates Airflow’s deferrable capabilities to enhance orchestrating dbt (https://github.com/dbt-labs/dbt-core) in production, with insights from recent contributions that introduced this functionality.
Key takeaways:
Deferrable Operators: How they work and why they’re ideal for long-running dbt tasks. Integrating with Cosmos: Refactoring and enhancements to enable deferrable behaviour across platforms. Performance Gains: Resource savings and task throughput improvements from deferrable execution. Challenges & Future Enhancements: Lessons learned, compatibility, and ideas for broader support. Whether orchestrating dbt models on a cloud warehouse or managing large-scale transformations, this session offers practical strategies to reduce resource contention and boost pipeline performance.


## Creating DuoFactory: An Orchestration Ecosystem with Airflow
[URL](https://airflowsummit.org/sessions/2025/creating-duofactory-a-orchestration-ecosystem-with-airflow/)  
Date: 2025-10-08 12:00:00
Summary:
Duolingo has built an internal tool DuoFactory to orchestrate AI generated content using Airflow. The tool has been used to generate example sentences per lesson, math exercises, and Duoradio lessons. The ecosystem is flexible for various company needs. Some of these use cases contain end to end generation where one click of a button generates content in app. We also have created a Workflow Builder to orchestrate and iterate on generative AI workflows by creating one-time DAG instances with a UI easy enough for non-engineers to use.


## Enterprise Auditing: "The Verifiable Data Pipeline"
[URL](https://airflowsummit.org/sessions/2025/enterprise-auditing/)  
Date: 2025-10-08 12:00:00
Summary:
This session will dive deep into leveraging the robust logging and audit capabilities of Google Cloud Platform, Cloud Composer and Apache Airflow to establish a fully transparent and verifiable data orchestration layer.
We’ll demonstrate how to track and attribute every change—from environment configuration to individual task execution—essential for meeting stringent enterprise governance, compliance, and auditing requirements.


## Airflow & Bigtop: Modernize and integrate time-proven OSS stack with Apache Airflow
[URL](https://airflowsummit.org/sessions/2025/airflow-bigtop-modernize-and-integrate-time-proven-oss-stack-with-apache-airflow/)  
Date: 2025-10-08 12:30:00
Summary:
Apache Bigtop is a time-proven open-source software stack for building data platform, which has been built around the Hadoop and Spark ecosystem since 2011. Its software composition has been changed during such a long period, and recently job scheduler is removed mainly due to the inactivity of its development. The speaker believes that Airflow perfectly fits into this gap and is proposing incorporating it in the Bigtop stack. This presentation will introduce how easily users can build a data platform with Bigtop including Airflow, and how Airflow can integrate those software with its wide range of providers and enterprise-readiness such as the Kerberos support.


## Airflow 3 UI is not enough? Add a Plugin!
[URL](https://airflowsummit.org/sessions/2025/airflow-3-ui-is-not-enough-add-a-plugin/)  
Date: 2025-10-08 12:30:00
Summary:
In Airflow 2 there was a plugin mechanism to extend the UI for new functions as well as be able to add hooks and other features.
As Airflow 3 rewrote the UI old Plugins were not working for all cases anymore. Airflow 3.1 now provides a re-vamped option to extend the UI with a new plugin schema in native React components and embedded iframes following AIP-68 definitions.
In this session we will provide an overview about capabilities and give some intro how you can roll-your-own.


## Do you trust Airflow with your money? (We do!)
[URL](https://airflowsummit.org/sessions/2025/do-you-trust-airflow-with-your-money-we-do/)  
Date: 2025-10-08 12:30:00
Summary:
Airflow is wonderfully, frustratingly complex - and so is global finance! Stripe has very specific needs all over the planet, and we have customized Airflow to adapt to the variety and rigor that we need to grow the GDP of the internet.
In this talk, you’ll learn:
How we support independent DAG change management for over 500 different teams running over 150k tasks.
How we’ve customized Airflow’s Kubernetes integration to comply with Stripe’s unique compliance requirements.


## LLMOps with Airflow 3.0 and the Airflow AI SDK
[URL](https://airflowsummit.org/sessions/2025/llmops-with-airflow-3-0-and-the-airflow-ai-sdk/)  
Date: 2025-10-08 12:30:00
Summary:
Airflow 3 brings several exciting new features that better support MLOps:
Native, intuitive backfills Removal of the unique execution date for dag runs Native support for event-driven scheduling These features, combined with the Airflow AI SDK, enable dag authors to easily build scalable, maintainable, and performant LLMOps pipelines.
In this talk, we’ll go through a series of workflows that use the Airflow AI SDK to empower Astronomer’s support staff to more quickly resolve problems faced by Astronomer’s customers.


## Airflow and Optimised Data Platform: Setup & Customisations
[URL](https://airflowsummit.org/sessions/2025/optimised-data-platform/)  
Date: 2025-10-08 14:00:00
Summary:
This workshop should be suitable for any Architect, Data Engineer or Devops aiming to build/enhance their internal Data Platform. At the end of this workshop you would have solid understanding of initial setup and ways to optimise further getting most out of the tool for your own organisation.

## Beyond Execution Dates: Empowering inference execution and hyper-parameter tuning with Airflow 3
[URL](https://airflowsummit.org/sessions/2025/beyond-execution-dates-empowering-inference-execution-and-hyper-parameter-tuning-with-airflow-3/)  
Date: 2025-10-08 14:00:00
Summary:
In legacy Airflow 2.x, each DAG run was tied to a unique “execution_date.” By removing this requirement, Airflow can now directly support a variety of new use cases, such as model training and generative AI inference, without the need for hacks and workarounds typically used by machine learning and AI engineers.
In this talk, we will delve into the significant advancements in Airflow 3 that enable GenAI and MLOps use cases, particularly through the changes outlined in AIP 83. We’ll cover key changes like the renaming of “execution_date” to “logical_date,” along with the allowance for it to be null, and the introduction of the new “run_after” field which provides a more meaningful mechanism for scheduling and sorting. Furthermore, we’ll discuss how Airflow 3 enables multiple parallel runs, empowering diverse triggering mechanisms and easing backfill logic with a real-world demo.


## Cloud Composer : Introduction into Advanced Features
[URL](https://airflowsummit.org/sessions/2025/cloud-composer/)  
Date: 2025-10-08 14:00:00
Summary:
Learn the latest features published within Cloud Composer which is a managed service for Apache Airflow on Google Cloud Platform.

## Event-Driven Airflow 3.0: Real-Time Orchestration with Pub/Sub
[URL](https://airflowsummit.org/sessions/2025/event-driven-airflow-3-0-real-time-orchestration-with-pub-sub/)  
Date: 2025-10-08 14:00:00
Summary:
Traditional time-based scheduling in Airflow can lead to inefficiencies and delays. With Airflow 3.0, we can now leverage native event-driven DAG execution, enabling workflows to trigger instantly when data arrives—eliminating polling-based sensors and rigid schedules. This talk explores real-time orchestration using Airflow 3.0 and Google Cloud Pub/Sub. We’ll showcase how to build an event-driven pipeline where DAGs automatically trigger as new data lands, ensuring faster and more efficient processing. Through a live demo, we’ll demonstrate how Airflow listens to Pub/Sub messages and dynamically triggers dbt transformations only when fresh data is available. This approach improves scalability, reduces costs, and enhances orchestration efficiency. Key Takeaways: How event-driven DAGs work vs. traditional scheduling, Best practices for integrating Airflow with Pub/Sub,Eliminating polling-based sensors for efficiency,Live demo: Event-driven pipeline with Airflow 3.0, Pub/Sub & dbt.


## GitHub's Airflow Journey: Lessons, Mistakes, and Insights
[URL](https://airflowsummit.org/sessions/2025/github-s-airflow-journey-lessons-mistakes-and-insights/)  
Date: 2025-10-08 14:00:00
Summary:
This session explores how GitHub uses Apache Airflow for efficient data engineering. We will share nearly 9 years of experiences, including lessons learnt, mistakes made, and the ways we reduced our on-call and engineering burden. We’ll demonstrate how we keep data flowing smoothly while continuously evolving Airflow and other components of our data platform, ensuring safety and reliability. The session will touch on how we migrate Airflow between cloud without user impact. We’ll also cover how we cut down the time from idea to running a DAG in production, despite our Airflow repo being among the top 15 by number of PRs within GitHub.


## Seamless Integration: Building Applications That Leverage Airflow's Database Migration Framework
[URL](https://airflowsummit.org/sessions/2025/seamless-integration-building-applications-that-leverage-airflow-s-database-migration-framework/)  
Date: 2025-10-08 14:00:00
Summary:
This session presents a comprehensive guide to building applications that integrate with Apache Airflow’s database migration system. We’ll explore how to harness Airflow’s robust Alembic-based migration toolchain to maintain schema compatibility between Airflow and custom applications, enabling developers to create solutions that evolve alongside the Airflow ecosystem without disruption.


## Airflow 3’s Trigger UI: Evolution of Params
[URL](https://airflowsummit.org/sessions/2025/airflow-3s-trigger-ui-evolution-of-params/)  
Date: 2025-10-08 14:30:00
Summary:
Are you looking to build slick, dynamic trigger forms for your DAGs? It all starts with mastering params.
Params are the gold standard for adding execution options to your DAGs, allowing you to create dynamic, user-friendly trigger forms with descriptions, validation, and now, with Airflow 3, bidirectional support for conf data!
In this talk, we’ll break down how to use params effectively, share best practices, and explore what’s new since the 2023 Airflow Summit talk (https://airflowsummit.org/sessions/2023/flexible-dag-trigger-forms-aip-50/). If you want to make DAG execution more flexible, intuitive, and powerful, this session is a must-attend!


## Beyond the bundle - evolving DAG parsing in Airflow 3
[URL](https://airflowsummit.org/sessions/2025/beyond-the-bundle-evolving-dag-parsing-in-airflow-3/)  
Date: 2025-10-08 14:30:00
Summary:
Airflow 3 made some great strides with AIP-66, introducing the concept of a DAG bundle. This successfully challenged one of the fundamental architectural limitations of original Airflow design of how DAGs are deployed, bringing the structure to something that often had to be operated as a pile of files in the past. However, we believe that this by no means should be the end of the road when it comes to making the DAG management easier, authoring more accessible to a broader audience, and integration with Data Agents smoother. We believe that the next step in Airflow’s evolution is in having a native option to break away from the necessity of having a real file in file systems on multiple components to have your DAG up and running. This is what we are hoping to achieve as part of AIP-85 - extendable DAG parsing control. In this talk I’d like to give a detailed overview of how we want to make it happen and show the examples of the valuable integrations we hope to unblock with it.


## Data Quality and Observability with Airflow
[URL](https://airflowsummit.org/sessions/2025/data-quality-and-observability-with-airflow/)  
Date: 2025-10-08 14:30:00
Summary:
Tekmetric is the largest cloud based auto shop management system in the United States. We process vast amounts of data from various integrations with internal and external systems. Data quality and governance are crucial for both our internal operations and the success of our customers.
We leverage multi-step data processing pipelines using AWS services and Airflow. While we utilize traditional data pipeline workflows to manage and move data, we go beyond standard orchestration. After data is processed, we apply tailored quality checks for schema validation, record completeness, freshness, duplication and more.


## UI Office Hours
[URL](https://airflowsummit.org/sessions/2025/ui-office-hours/)  
Date: 2025-10-08 14:30:00
Summary:
Join this live demo of the Airflow UI while we answer your questions and discuss your ideas on how to improve the experience.


## Building Airflow 3 setups resilient to zonal/regional down events, ready for Disaster Recovery event
[URL](https://airflowsummit.org/sessions/2025/building-airflow-3-setups-resilient-to-zonal-regional-down-events-ready-for-disaster-recovery-event/)  
Date: 2025-10-08 15:00:00
Summary:
Want to be resilient to any zonal/regional down events when building Airflow in a cloud environment? Unforeseen disruptions in cloud infrastructure, whether isolated to specific zones or impacting entire regions, pose a tangible threat to the continuous operation of critical data workflows managed by Airflow. These outages, though often technical in nature, translate directly into real-world consequences, potentially causing interruptions in essential services, delays in crucial information delivery, and ultimately impacting the reliability and efficiency of various operational processes that businesses and individuals depend upon daily. The inability to process data reliably due to infrastructure instability can cascade into tangible setbacks across diverse sectors, highlighting the urgent need for resilient and robust Airflow deployments.


## How Airflow Runs The Weather
[URL](https://airflowsummit.org/sessions/2025/how-airflow-runs-the-weather/)  
Date: 2025-10-08 15:00:00
Summary:
Forecasting the weather and air quality is a logistical challenge. Numerical simulations are complex, resource-hungry, and sometimes fail without warning. Yet, our clients depend on accurate forecasts delivered daily and on time. At the heart of this operation is Airflow: the orchestration engine that keeps everything running.
In this session, we’ll dive into the world behind weather and air quality forecasts. In particular, we’ll explore:
The atmospheric modeling pipeline, to understand the unique demands it places on infrastructure How we use Airflow to orchestrate complex simulations reliably and at scale, to inspire new ways of managing time-critical, compute-heavy workflows. Our integration of Airflow with a high-performance computing (HPC) environment using Slurm, to run resource-intensive workloads efficiently in bare metal machines. At Meteosim we are experts on weather and air quality intelligence. With projects in over 80 countries, we support decision-making in industries where weather and air quality matter most: from daily operations to long-term sustainability.


## Simplifying Data Lineage: How OpenLineage Empowers Airflow and Beyond
[URL](https://airflowsummit.org/sessions/2025/simplifying-data-lineage-how-openlineage-empowers-airflow-and-beyond/)  
Date: 2025-10-08 15:00:00
Summary:
OpenLineage has simplified collecting lineage metadata across the data ecosystem by standardizing its representation in an extensible model. It enabled a whole ecosystem improving data pipeline reliability and ease of troubleshooting in production environments. In this talk, we’ll briefly introduce the OpenLineage model and explore how this metadata is collected from Airflow, Spark, dbt, and Flink. We’ll demonstrate how to extract valuable insights and outline practical benefits and common challenges when building ingestion, processing and storage for OpenLineage data. We will also briefly show how OpenLineage events can be used to observe data pipelines exhastively and the benefits that brings.


## Why Data Teams Keep Reinventing the Wheel: The Struggle for Code Reuse in the Data Transformation La
[URL](https://airflowsummit.org/sessions/2025/why-data-teams-keep-reinventing-the-wheel-the-struggle-for-code-reuse-in-the-data-transformation-la/)  
Date: 2025-10-08 15:00:00
Summary:
Data teams have a bad habit: reinventing the wheel. Despite the explosion of open-source tooling, best practices, and managed services, teams still find themselves building bespoke data platforms from scratch—often hitting the same roadblocks as those before them. Why does this keep happening, and more importantly, how can we break the cycle?
In this talk, we’ll unpack the key reasons data teams default to building rather than adopting, from technical nuances to cultural and organizational dynamics. We’ll discuss why fragmentation in the modern data stack, the pressure to “own” infrastructure, and the allure of in-house solutions make this problem so persistent.


## Assets: Past, Present, Future
[URL](https://airflowsummit.org/sessions/2025/assets-past-present-future/)  
Date: 2025-10-08 15:45:00
Summary:
Airflow Asset originated from data lineage and evolved into its current state, being used as a scheduling concept (data-aware, event-based scheduling). It has even more potential. This talk discusses how other parts of Airflow, namely Connection and Object Storage, contain concepts related to Asset, and we can tie them all together to make task authoring flow even more naturally.
Planned topics:
Brief history on Asset and related constructs. Current state of Asset concepts. Inlets, anyone? Finding inspiration from Pydantic et al. My next step for Asset. 

## From Legacy to Leading Edge: How Airflow Migration Unlocked Cross-Team Business Value
[URL](https://airflowsummit.org/sessions/2025/from-legacy-to-leading-edge-how-airflow-migration-unlocked-cross-team-business-value/)  
Date: 2025-10-08 15:45:00
Summary:
At TrueCar, migrating hundreds of legacy workflows from in-house orchestration tools to Apache Airflow required key technical decisions that transformed our data platform architecture and organizational capabilities.
We consolidated individual chained tasks into optimized DAGs leveraging native Airflow functionality to trigger compute across cloud environments. A crucial breakthrough was developing DAG generators to scale migration—essential for efficiently migrating hundreds of workflows while maintaining consistency.
By decoupling orchestration from compute, we gained flexibility to select optimal tools for specific outcomes—programmatic processing, analytics, batch jobs, or AI/ML pipelines. This resulted in cost reductions, performance improvements, and team agility. We also gained unprecedented visibility into DAG performance and dependency patterns previously invisible across fragmented systems.


## Seamless Migration: Leveraging Ruff for a Smooth Transition from Airflow 2 to Airflow 3
[URL](https://airflowsummit.org/sessions/2025/seamless-migration-leveraging-ruff-for-a-smooth-transition-from-airflow-2-to-airflow-3/)  
Date: 2025-10-08 15:45:00
Summary:
Migrating from Airflow 2 to the newly released Airflow 3 may seem intimidating due to numerous breaking changes and the introduction of new features. Although a backward compatibility layer has been implemented and most of the existing dags should work fine, some features—such as subdags and execution_date—have been removed based on community consensus.
To support this transition, we worked with Ruff to establish rules that automatically identify removed or deprecated features and even assist in fixing them. In this presentation, I will outline our current Ruff features, the migration rules from Airflow 2 to 3, and how this experience opens the door for us to promote best practices in Airflow through Ruff in the future.


## Semiconductor (Chip) Design Workflow Orchestration with Airflow
[URL](https://airflowsummit.org/sessions/2025/semiconductor-chip-design-workflow-orchestration-with-airflow/)  
Date: 2025-10-08 15:45:00
Summary:
The design of Qualcomm’s Snapdragon System-On-Chip (SoCs) involves several hundred complex workflows orchestrated across multiple data centers, taking the design from RTL to GDS. In the Snapdragon Oryon Custom CPU team, we introduced Airflow about 2 years ago to orchestrate design, verification, emulation, CI/CD, and physical implementation of our CPUs.
Use Case: • Standardization and Templatization: We standardize and templatize common workflows, allowing designers to verify their designs by customizing YAML parameters. • Custom Shell Operators: We created custom shell operators (tcshrc) to source project environments and work with internal tooling. • Smart Retries: We use pre/post-execute hooks to trigger smart retries on failure. • Dynamic Celery Workers: We auto-create Celery workers on the fly on our High-Performance Compute (HPC) clusters to launch and manage Electronic Design Automation (EDA) workloads. • Hybrid Executor Strategy: We use a hybrid executor strategy (CeleryExecutor and EdgeExecutor) to orchestrate tasks across multiple data centers. • EdgeExecutor for Remote Testing: We leverage EdgeExecutor to access post-silicon hardware in remote locations.


## Apache Airflow 3.0 - Bad vs. Best Practices In Production
[URL](https://airflowsummit.org/sessions/2025/apache-airflow-3-0-bad-vs-best-practices-in-production/)  
Date: 2025-10-08 16:15:00
Summary:
The general-purpose nature of Airflow has always left us questioning, “Is this the right way”? While the existing resources and community cover them, the new Airflow releases always leave us wondering if there is more . This talk reveals how 3.0’s innovations redefine best practices, building production-ready data platforms.
• Dag Development - Future-proof your dags without compromising on Fundamentals • Modern Pipelines: How to best incorporate new Airflow features • Infrastructure: Leveraging 3.0’s Service-Oriented Architecture and Edge Executor • Teams & Responsibilities: Streamlined operations with the new split CLI and improved UI. • Monitoring & Observability: Building fail-proof pipelines


## How Airflow solves the coordination of decentralised teams at Vinted
[URL](https://airflowsummit.org/sessions/2025/how-airflow-solves-the-coordination-of-decentralised-teams-vinted/)  
Date: 2025-10-08 16:15:00
Summary:
Vinted is the biggest second-hand marketplace in Europe with multiple business verticals. Our data ecosystem has over 20 decentralized teams responsible for generating, transforming, and building Data Products from petabytes of data. This creates a daring environment where inter-team dependencies, varied expertise with scheduling tools, and diverse use cases need to be managed efficiently. To tackle these challenges, we have centralized our approach by leveraging Apache Airflow to orchestrate data dependencies across teams.


## Purple is the new green: harnessing deferrable operators to improve performance & reduce costs
[URL](https://airflowsummit.org/sessions/2025/purple-is-the-new-green-harnessing-deferrable-operators-to-improve-performance-reduce-costs/)  
Date: 2025-10-08 16:15:00
Summary:
Airflow’s traditional execution model often leads to wasted resources: worker nodes sitting idle, waiting on external systems. At Wix, we tackled this inefficiency head-on by refactoring our in-house operators to support Airflow’s deferrable execution model.
Join us on a walk through Wix’s journey to a more efficient Airflow setup, from identifying bottlenecks to implementing deferrable operators and reaping their benefits. We’ll share the alternatives considered, the refactoring process, and how the team seamlessly integrated deferrable execution with no disruption to data engineers’ workflows.


## Behind the Scenes: How We Tested Airflow 3 for Stability and Reliability
[URL](https://airflowsummit.org/sessions/2025/behind-the-scenes-how-we-tested-airflow-3-for-stability-and-reliability/)  
Date: 2025-10-08 16:45:00
Summary:
Ensuring the stability of a major release like Airflow 3 required extensive testing across multiple dimensions. In this session, we will dive into the testing strategies and validation techniques used to guarantee a smooth rollout. From unit and integration tests to real-world DAG validations, this talk will cover the challenges faced, key learnings, and best practices for testing Airflow. Whether you’re a contributor, QA engineer, or Airflow user preparing for migration, this session will offer valuable takeaways to improve your own testing approach.


## From Centrailization to Autonomy: Managing Airflow Pipeline through Multi-Tenancy
[URL](https://airflowsummit.org/sessions/2025/from-centrailization-to-autonomy/)  
Date: 2025-10-08 16:45:00
Summary:
At the enterprise level, managing Airflow deployments across multiple teams can become complex, leading to bottlenecks and slowed development cycles. We will share our journey of decentralizing Airflow repositories to empower data engineering teams with multi-tenancy, clean folder structures, and streamlined DevOps processes.
We dive into how restructuring our Airflow architecture and utilizing repository templates allowed teams to generate new data pipelines effortlessly. This approach enables engineers to focus on business logic without worrying about underlying Airflow configurations. By automating deployments and reducing manual errors through CI/CD pipelines, we minimized operational overhead.


## Run Airflow tasks on your coffee machine
[URL](https://airflowsummit.org/sessions/2025/run-airflow-tasks-on-your-coffee-machine/)  
Date: 2025-10-08 16:45:00
Summary:
Airflow 3 comes with two new features: Edge execution and the task SDK. Powered by a HTTP API, these make it possible to write and execute Airflow tasks in any language from anywhere.
In this session I will explain some of the APIs needed and show how to interact with them based on an embedded toy worker written in Rust and running on an ESP32-C3. Furthermore I will provide practical tips on writing your own edge worker and how to develop against a running instance of Airflow.


## Airflow as a Platform for Agentic AI Digital Products Within Enterprises
[URL](https://airflowsummit.org/sessions/2025/airflow-as-a-platform-for-agentic-ai/)  
Date: 2025-10-09 09:15:00
Summary:
In this keynote, Peeyush Rai and Vikram Koka will be walking through how Airflow is being used as part of a Agentic AI platform servicing insurance companies, which runs on all the major public clouds, leveraging models from Open AI, Google (Gemini), AWS (Claude and Bedrock).
This talk walks through the details of the actual end user business workflow including gathering relevant financial data to make a decision, as well as the tricky challenge of handling AI hallucinations, with new Airflow capabilities such as “Human in the loop”.


## Becoming an Apache Airflow Committer from 0
[URL](https://airflowsummit.org/sessions/2025/becoming-an-apache-airflow-committer-from-0/)  
Date: 2025-10-09 10:30:00
Summary:
How a Complete Beginner in Data Engineering / Junior Computer Science Student Became an Apache Airflow Committer in Just 5 Months—With 70+ PRs and 300 Hours of Contributions
This talk is aimed at those who are still hesitant about contributing to Apache Airflow. I hope to inspire and encourage anyone to take the first step and start their journey in open-source—let’s build together!


## Bridging Data Pipelines and Business Applications with Airflow and Control-M
[URL](https://airflowsummit.org/sessions/2025/control-m/)  
Date: 2025-10-09 10:30:00
Summary:
Learn how Control-M integrates with Airflow to orchestrate end-to-end workflows that include upstream and downstream enterprise systems like Supply Chain and Billing. Gain visibility, reliability, and seamless coordination across your data pipelines and the business operations they support.

## Enhancing Small Retailer Visibility: Machine Learning Pipelines with Apache Airflow
[URL](https://airflowsummit.org/sessions/2025/enhancing-small-retailer-visibility/)  
Date: 2025-10-09 10:30:00
Summary:
Small retailers often lack the data visibility that larger companies rely on for decision-making. In this session, we’ll dive into how Apache Airflow powers end-to-end machine learning pipelines that process inventory and sales data, enabling retailers and suppliers to gain valuable industry insights. We’ll cover feature engineering, model training, and automated inference workflows, along with strategies for handling messy, incomplete retail data. We will discuss how Airflow enables scalable ML-driven insights that improve demand forecasting, product categorization, and supply chain optimization.


## Get Certified: DAG Authoring for Apache Airflow 3
[URL](https://airflowsummit.org/sessions/2025/get-certified-dag-authoring-for-apache-airflow-3/)  
Date: 2025-10-09 10:30:00
Summary:
The DAG Authoring for Apache Airflow certification validates your expertise in advanced Airflow concepts and demonstrates your ability to build production-grade data pipelines.

## Orchestrating MLOps and Data Transformation at EDB with Airflow
[URL](https://airflowsummit.org/sessions/2025/orchestrating-mlops-and-data-transformation-at-edb-with-airflow/)  
Date: 2025-10-09 10:30:00
Summary:
This talk explores EDB’s journey from siloed reporting to a unified data platform, powered by Airflow. We’ll delve into the architectural evolution, showcasing how Airflow orchestrates a diverse range of use cases, from Analytics Engineering to complex MLOps pipelines.
Learn how EDB leverages Airflow and Cosmos to integrate dbt for robust data transformations, ensuring data quality and consistency.
We’ll provide a detailed case study of our MLOps implementation, demonstrating how Airflow manages training, inference, and model monitoring pipelines for Azure Machine Learning models.


## Scaling ML Infrastructure: Lessons from Building Distributed Systems
[URL](https://airflowsummit.org/sessions/2025/scaling-ml-infrastructure/)  
Date: 2025-10-09 10:30:00
Summary:
In today’s data-driven world, scalable ML infrastructure is mission-critical. As ML workloads grow, orchestration tools like Apache Airflow become essential for managing pipelines, training, deployment, and observability. In this talk, I’ll share lessons from building distributed ML systems across cloud platforms, including GPU-based training and AI-powered healthcare. We’ll cover patterns for scaling Airflow DAGs, integrating telemetry and auto-healing, and aligning cross-functional teams. Whether you’re launching your first pipeline or managing ML at scale, you’ll gain practical strategies to make Airflow the backbone of your ML infrastructure.


## Building an Airflow Center of Excellence: Lessons from the Frontlines
[URL](https://airflowsummit.org/sessions/2025/building-an-airflow-center-of-excellence-lessons-from-the-frontlines/)  
Date: 2025-10-09 11:00:00
Summary:
As organizations scale their data infrastructure, Apache Airflow becomes a mission-critical component for orchestrating workflows efficiently. But scaling Airflow successfully isn’t just about running pipelines—it’s about building a Center of Excellence (CoE) that empowers teams with the right strategy, best practices, and long-term enablement. Join Jon Leek and Michelle Winters as they share their experiences helping customers design and implement Airflow Centers of Excellence. They’ll walk through real-world challenges, best practices, and the structured approach Astronomer takes to ensure teams have the right plan, resources, and support to succeed. Whether you’re just starting with Airflow or looking to optimize and scale your workflows, this session will give you a proven framework to build a sustainable Airflow Center of Excellence within your organization. 🚀


## How Pinterest Uses Ai to Empower Airflow Users for Troubleshooting
[URL](https://airflowsummit.org/sessions/2025/how-pinterest-uses-ai-to-empower-airflow-users-for-troubleshooting/)  
Date: 2025-10-09 11:00:00
Summary:
At Pinterest, there are over 10,000 DAGs supporting various use cases across different teams and roles. With this scale and diversity, user support has been an ongoing challenge to unlock productivity. As Airflow increasingly serves as a user interface to a variety of data and ML infrastructure behind the scenes, it’s common for issues from multiple areas to surface in Airflow, making triage and troubleshooting a challenge.
In this session, we will discuss the scale of the problem we are facing, how we have addressed it so far, and how we are introducing LLM AI to help solve this problem.


## Your privacy or our progress: rethinking telemetry in Airflow
[URL](https://airflowsummit.org/sessions/2025/your-privacy-or-our-progress-rethinking-telemetry-in-airflow/)  
Date: 2025-10-09 11:00:00
Summary:
We face a paradox: we could use usage data to build better software, but collecting that data seems to contradict the very principles of user freedom that open source represents. Apache Airflow’s current telemetry - already purged - system has become a battleground for this conflict, with some users voicing concerns over privacy while maintainers struggle to make informed decisions without data. What can we do to strike the right balance?


## Custom Operators in Action: A Guide to Extending Airflow's Capabilities
[URL](https://airflowsummit.org/sessions/2025/custom-operators-in-action/)  
Date: 2025-10-09 11:30:00
Summary:
Custom operators are the secret weapon for solving Airflow’s unique & challenging orchestration problems.
This session will cover:
When to build custom operators vs. using existing solutions Architecture patterns for creating maintainable, reusable operators Live coding demonstration: Building a custom operator from scratch Real-world examples: How custom operators solve specific business challenges Through practical code examples and architecture patterns, attendees will walk away with the knowledge to implement custom operators that enhance their Airflow deployments.


## Data & AI Orchestration at GoDaddy
[URL](https://airflowsummit.org/sessions/2025/data-and-ai-orchestration-at-godaddy/)  
Date: 2025-10-09 11:30:00
Summary:
As the adoption of Airflow increases within large enterprises to orchestrate their data pipelines, more than one team needs to create, manage, and run their workflows in isolation. With multi-tenancy not yet supported natively in Airflow, customers are adopting alternate ways to enable multiple teams to share infrastructure. In this session, we will explore how GoDaddy uses MWAA to build a Single Pane Airflow setup for multiple teams with a common observability platform, and how this foundation enables orchestration expansion beyond data workflows to AI workflows as well. We’ll discuss our roadmap for leveraging upcoming Airflow 3 features, including the task execution API for enhanced workflow management and DAG versioning capabilities for comprehensive auditing and governance. This session will help attendees gain insights into the use case, the solution architecture, implementation challenges and benefits, and our strategic vision for unified orchestration across data and AI workloads.


## From Oops to Secure Ops: Self-Hosted AI for Airflow Failure Diagnosis
[URL](https://airflowsummit.org/sessions/2025/from-oops-to-secure-ops-self-hosted-ai-for-airflow-failure-diagnosis/)  
Date: 2025-10-09 11:30:00
Summary:
Last year, ‘From Oops to Ops’ showed how AI-powered failure analysis could help diagnose why Airflow tasks fail. But do we really need large, expensive cloud-based AI models to answer simple diagnostic questions? Relying on external AI APIs introduces privacy risks, unpredictable costs, and latency, often without clear benefits for this use case.
With the rise of distilled, open-source models, self-hosted failure analysis is now a practical alternative. This talk will explore how to deploy an AI service on infrastructure you control, compare cost, speed, and accuracy between OpenAI’s API and self-hosted models, and showcase a live demo of AI-powered task failure diagnosis using DeepSeek and Llama—running without external dependencies to keep data private and costs predictable.


## Orchestrating AI Knowledge Bases with Apache Airflow
[URL](https://airflowsummit.org/sessions/2025/orchestrating-ai-knowledge-bases-with-apache-airflow/)  
Date: 2025-10-09 11:30:00
Summary:
In the age of Generative AI, knowledge bases are the backbone of intelligent systems, enabling them to deliver accurate and context-aware responses. But how do you ensure that these knowledge bases remain up-to-date and relevant in a rapidly changing world? Enter Apache Airflow, a robust orchestration tool that streamlines the automation of data workflows.
This talk will explore how Airflow can be leveraged to manage and update AI knowledge bases across multiple data sources. We’ll dive into the architecture, demonstrate how Airflow enables efficient data extraction, transformation, and loading (ETL), and share insights on tackling challenges like data consistency, scheduling, and scalability.


## 5 Simple Strategies To Enhance Your DAGs For Data Processing
[URL](https://airflowsummit.org/sessions/2025/5-simple-strategies-to-enhance-your-dags-for-data-processing/)  
Date: 2025-10-09 12:00:00
Summary:
Take your DAGs in Apache Airflow to the next level? This is an insightful session where we’ll uncover 5 transformative strategies to enhance your data workflows. Whether you’re a data engineering pro or just getting started, this presentation is packed with practical tips and actionable insights that you can apply right away.
We’ll dive into the magic of using powerful libraries like Pandas, share techniques to trim down data volumes for faster processing, and highlight the importance of modularizing your code for easier maintenance. Plus, you’ll discover efficient ways to monitor and debug your DAGs, and how to make the most of Airflow’s built-in features.


## Airflow Uses in an on-prem Research Setting
[URL](https://airflowsummit.org/sessions/2025/airflow-uses-in-an-on-prem-research-setting/)  
Date: 2025-10-09 12:00:00
Summary:
KP Division of Research uses Airflow as a central technology for integrating diverse technologies in an agile setting. We wish to present a set of use-cases for AI/ML workloads, including imaging analysis (tissue segmentation, mammography), NLP (early identification of psychosis), LLM processing (identification of vessel diameter from radiological impressions), and other large data processing tasks. We create these “short-lived” project workflows to accomplish specific aims, and then may never run the job again, so leveraging generalized patterns are crucial to quickly implementing these jobs. Our Advanced Computational Infrastructure is comprised of multiple Kubernetes clusters, and we use Airflow to democratize the use of our batch level resources in those clusters. We use Airflow form-based parameters to deploy pods running R and Python scripts where generalized parameters are injected into scripts that follow internal programming patterns. Finally, we also leverage Airflow to create headless services inside Kubernetes for large computational workloads (Spark & H2O) that subsequent pods consume ephemerally.


## Enhancing Airflow REST API: From Basic Integration to Enterprise Scale
[URL](https://airflowsummit.org/sessions/2025/enhancing-airflow-rest-api-from-basic-integration-to-enterprise-scale/)  
Date: 2025-10-09 12:00:00
Summary:
Apache Airflow’s REST API has evolved to support diverse orchestration needs, with managed services like MWAA introducing custom enhancements. One such feature, InvokeRestApi, enables dynamic interactions with external services while maintaining Airflow’s core orchestration capabilities.
In this talk, we will explore the architectural design behind InvokeRestApi, detailing how it enhances API-driven workflows. Beyond the architecture, we’ll share key challenges and learnings from implementing and scaling Airflow’s REST API in production environments. Topics include authentication, performance considerations, error handling, and best practices for integrating external APIs efficiently.


## Using Apache Airflow with Trino for (almost) all your data problems
[URL](https://airflowsummit.org/sessions/2025/using-apache-airflow-with-trino-for-almost-all-your-data-problems/)  
Date: 2025-10-09 12:00:00
Summary:
Trino is incredibly effective at enabling users to extract insights quickly and effectively from large amount of data located in dispersed and heterogeneous federated data systems.
However, some business data problems are more complex than interactive analytics use cases, and are best broken down into a sequence of interdependent steps, a.k.a. a workflow. For these use cases, dedicated software is often required in order to schedule and manage these processes with a principled approach.


## Common provider abstractions: Key for multi-cloud data handling
[URL](https://airflowsummit.org/sessions/2025/common-provider-abstractions-key-for-multi-cloud-data-handling/)  
Date: 2025-10-09 12:30:00
Summary:
Enterprises want the flexibility to operate across multiple clouds, whether to optimize costs, improve resiliency, to avoid vendor lock-in, or for data sovereignty. But for developers, that flexibility usually comes at the cost of extra complexity and redundant code. The goal here is simple: write once, run anywhere, with minimum boilerplate. In Apache Airflow, we’ve already begun tackling this problem with abstractions like Common-SQL, which lets you write database queries once and run them on 20+ databases, from Snowflake to Postgres to SQLite to SAP HANA. Similarly, Common-IO standardizes cloud blob storage interactions across all public clouds. With Airflow 3.0, we are pushing this further by introducing a Common Message Bus provider, which is an abstraction, initially supporting Amazon SQS and expanding to Google PubSub and Apache Kafka soon after. We expect additional implementations such as Amazon Kinesis and Managed Kafka over time.


## Dynamic DAGs and Data Quality using DAGFactory
[URL](https://airflowsummit.org/sessions/2025/dynamic-dags-data-quality/)  
Date: 2025-10-09 12:30:00
Summary:
We have a similar pattern of DAGs running for different data quality dimensions like accuracy, timeliness, & completeness. To do this again and again, we would be duplicating and potentially introducing human error while doing copy paste of code or making people write same code again.
To solve for this, we are doing few things:
Run DAGs via DagFactory to dynamically generate DAGs using just some YAML code for all the steps we want to run in our DQ checks. Hide this behind a UI which is hooked to github PR open step, now the user just provides some inputs or selects from dropdown in UI and a YAML DAG is generated for them. This highlights the potential for DAGFactory to hide Airflow Python code from users and make it more accessible to Data Analysts and Business Intelligence along with normal Software Engg, along with reducing human error. YAML is the perfect format to be able to generate code, create a PR and DagFactory is the perfect fir for that. All of this is running in GCP Cloud Composer.


## Scaling and Unifying Multiple Airflow Instances with Orchestration Frederator
[URL](https://airflowsummit.org/sessions/2025/scaling-and-unifying-multiple-airflow-instances-with-orchestration-frederator/)  
Date: 2025-10-09 12:30:00
Summary:
In large organizations, multiple Apache Airflow instances often arise organically—driven by team-specific needs, distinct use cases, or tiered workloads. This fragmentation introduces complexity, operational overhead, and higher infrastructure costs. To address these challenges, we developed the “Orchestration Frederator,” a solution designed to unify and horizontally scale multiple Airflow deployments seamlessly.
This session will detail our journey in implementing Orchestration Frederator, highlighting how we achieved:
Horizontal Scalability: Seamlessly scaling Airflow across multiple instances without operational overhead.


## Transforming Data Engineering: Achieving Efficiency and Ease with an Intuitive Orchestration Solution
[URL](https://airflowsummit.org/sessions/2025/transforming-data-engineering-achieving-efficiency-and-ease-with-an-intuitive-orchestration-solution/)  
Date: 2025-10-09 12:30:00
Summary:
In the rapidly evolving field of data engineering and data science, efficiency and ease of use are crucial. Our innovative solution offers a user-friendly interface to manage and schedule custom PySpark, PySQL, Python, and SQL code, streamlining the process from development to production. Using Airflow at the backend, this tool eliminates the complexities of infrastructure management, version control, CI/CD processes, and workflow orchestration.The intuitive UI allows users to upload code, configure job parameters, and set schedules effortlessly, without the need for additional scripting or coding. Additionally, users have the flexibility to bring their own custom artifactory solution and run their code. In summary, our solution significantly enhances the orchestration and scheduling of custom code, breaking down traditional barriers and empowering organizations to maximize their data’s potential and drive innovation efficiently. Whether you are an individual data scientist or part of a large data engineering team, this tool provides the resources needed to streamline your workflow and achieve your goals faster than ever before.


## A Decade in Data Engineering - Lessons Realities and Where We Go From Here
[URL](https://airflowsummit.org/sessions/2025/a-decade-in-data-engineering-lessons-realities-and-where-we-go-from-here/)  
Date: 2025-10-09 14:00:00
Summary:
There was a post on the data engineering subreddit recently that discussed how difficult it is to keep up with the data engineering world.
Did you learn Hadoop, great we are on Snowflake, BigQuery and Databricks now.
Just learned Airflow, well now we have Airflow 3.0.
And the list goes on.
But what doesn’t change, and what have been the lessons over the past decade. That’s what I’ll be covering in this talk. Real lessons and realities that come up time and time again whether you’re working for a start-up or a large enterprise.


## From Cron to Data-Aware: Evolving Airflow Scheduling at Scale
[URL](https://airflowsummit.org/sessions/2025/from-cron-to-data-aware/)  
Date: 2025-10-09 14:00:00
Summary:
As data platforms grow in complexity, so do the orchestration needs behind them. Time-based (cron) scheduling has long been the default in Airflow, but dataset-based scheduling promises a more data-aware, efficient alternative. In this session, I’ll share lessons learned from operating Airflow at scale—supporting thousands of DAGs across teams with varied use cases, from simple ETL to complex ML workflows. We’ll explore when dataset scheduling makes sense, the challenges it introduces, and how to evolve your DAG design and platform architecture to make the most of it. Whether you’re migrating legacy workflows or designing new ones, this talk will help you evaluate the right scheduling model for your needs.


## Implementing Operations Research Problems with Apache Airflow: From Modelling to Production
[URL](https://airflowsummit.org/sessions/2025/implementing-operations-research-problems-with-apache-airflow-from-modelling-to-production/)  
Date: 2025-10-09 14:00:00
Summary:
Hands-on session where attendees will gain experience creating DAGs to define and manage workflows for classical operations research problems.

## Lessons learned from migrating to Airflow @ LinkedIn
[URL](https://airflowsummit.org/sessions/2025/lessons-learned-from-migrating-to-airflow-li-scale/)  
Date: 2025-10-09 14:00:00
Summary:
At LinkedIn, our data pipelines process exabytes of data, with our offline infrastructure executing 300K ETL workflows daily and 10K concurrent executions. Historically, these workloads ran on our legacy system, Azkaban, which faced UX, scalability, and operational challenges. To modernize our infra, we built a managed Airflow service, leveraging its enhanced developer & operator experience, rich feature set, and strong OSS community support. That initiated LinkedIn’s largest-ever infrastructure migration—transitioning thousands of legacy workflows to Airflow.


## Productionising dbt-core with Airflow
[URL](https://airflowsummit.org/sessions/2025/productionising-dbt-core/)  
Date: 2025-10-09 14:00:00
Summary:
This workshop will cover a step-by-step guide to Cosmos, an open-source package that helps you quickly run your dbt Core projects as Airflow DAGs and Task Groups.

## Task failures troubleshooting based on Airflow & Kubernetes signals
[URL](https://airflowsummit.org/sessions/2025/task-failures-troubleshooting-based-on-airflow-kubernetes-signals/)  
Date: 2025-10-09 14:00:00
Summary:
Per Airflow community survey, Kubernetes is the most popular compute platform used to run Airflow and when run on Kubernetes, Airflow gains, out of the box, lots of benefits like monitoring, reliability, ease of deployment, scalability and autoscaling. On the other hand, running Airflow on Kubernetes means running a sophisticated distributed system on another distributed system which makes troubleshooting of Airflow tasks and DAGs failures harder.
This session tackles that bottleneck head-on, introducing a practical approach to building an automated diagnostic pipeline for Airflow on Kubernetes. Imagine offloading tedious investigations to a system that, on task failure, automatically collects and correlates key signals from Kubernetes components (linking Airflow tasks to specific Pods and their events), KubernetesGKE monitoring, and relevant logs—pinpointing root causes and suggesting actionable fixes.


## Your first Apache Airflow Contribution
[URL](https://airflowsummit.org/sessions/2025/your-first-apache-airflow-contribution/)  
Date: 2025-10-09 14:00:00
Summary:
Whether you’re writing code, enhancing documentation, or offering feedback, there’s a place for you. Let’s get started and see your name among Airflow contributors!"

## Applying Airflow to drive the digital workforce in the Enterprise
[URL](https://airflowsummit.org/sessions/2025/applying-airflow-to-drive-the-digital-workforce-in-the-enterprise/)  
Date: 2025-10-09 14:30:00
Summary:
Red Hat’s unified data and AI platform relies on Apache Airflow for orchestration, alongside Snowflake, Fivetran, and Atlan. The platform prioritizes building a dependable data foundation, recognizing that effective AI depends on quality data. Airflow was selected for its predictability, extensive connectivity, reliability, and scalability.
The platform now supports business analytics, transitioning from ETL to ELT processes. This has resulted in a remarkable improvement in how we make data available for business decisions.


## Learn from Deutsche Bank: Using Apache Airflow in Regulated Environments
[URL](https://airflowsummit.org/sessions/2025/learn-from-deutsche-bank-using-apache-airflow-in-regulated-environments/)  
Date: 2025-10-09 14:30:00
Summary:
Operating within the stringent regulatory landscape of Corporate Banking, Deutsche Bank relies heavily on robust data orchestration. This session explores how Deutsche Bank’s Corporate Bank leverages Apache Airflow across diverse environments, including both on-premises infrastructure and cloud platforms. Discover their approach to managing critical data & analytics workflows, encompassing areas like regulatory reporting, data integration and complex data processing pipelines. Gain insights into the architectural patterns and operational best practices employed to ensure compliance, security, and scalability when running Airflow at scale in a highly regulated, hybrid setting.


## Navigating Secure and Cost-Efficient Flink Batch on Kubernetes with Airflow
[URL](https://airflowsummit.org/sessions/2025/navigating-secure-and-cost-efficient-flink-batch-on-kubernetes-with-airflow/)  
Date: 2025-10-09 14:30:00
Summary:
At Yahoo, we built a secure, scalable, and cost-efficient batch processing platform using Amazon MWAA to orchestrate Apache Flink jobs on EKS, managed by the Flink Kubernetes Operator. This setup enables dynamic job orchestration while meeting strict enterprise compliance standards.
In this session, we’ll share how Airflow DAGs:
Dynamically launch, monitor, and clean up isolated Flink clusters per batch job, improving resource efficiency.
Securely fetch EKS kubeconfig, submit FlinkDeployment CRDs using FlinkKubernetesOperator, and poll job status using Airflow sensors.


## Simplifying Data Management with DAG Factory
[URL](https://airflowsummit.org/sessions/2025/simplifying-data-management-with-dag-factory/)  
Date: 2025-10-09 14:30:00
Summary:
At OLX, we connect millions of people daily through our online marketplace while relying on robust data pipelines. In this talk, we explore how the DAG Factory concept elevates data governance, lineage, and discovery by centralizing operator logic and restricting direct DAG creation. This approach enforces code quality, optimizes resources, maintains infrastructure hygiene and enables smooth version upgrades. We then leverage consistent naming conventions in Airflow to build targeted namespaces, aligning teams with global policies while preserving autonomy. Integrating external tools like AWS Lake Formation and Open Metadata further unifies governance, making it straightforward to manage and secure data. This is critical when handling hundreds or even thousands of active DAGs. If the idea of storing 1,600 pipelines in one folder seems overwhelming, join us to learn how the DAG Factory concept simplifies pipeline management. We’ll also share insights from OLX, highlighting how thoughtful design fosters oversight, efficiency, and discoverability across diverse use cases.


## AWS Lambda Executor: The Speed of Local Execution with the Advantages of Remote
[URL](https://airflowsummit.org/sessions/2025/aws-lambda-executor-the-speed-of-local-execution-with-the-advantages-of-remote/)  
Date: 2025-10-09 15:00:00
Summary:
Apache Airflow’s executor landscape has traditionally presented users with a clear trade-off: choose either the speed of local execution or the scalability, isolation and configurability of remote execution. The AWS Lambda Executor introduces a new paradigm that bridges this gap, offering near-local execution speeds with the benefits of remote containerization.
This talk will begin with a brief overview of Airflow’s executors, how they work and what they are responsible for, highlighting the compromises between different executors. We will explore the emerging niche for fast, yet remote execution and demonstrate how the AWS Lambda Executor fills this space. We will also address practical considerations when using such an executor, such as working within Lambda’s 15 minute execution limit, and how to mitigate this using multi-executor configuration.


## DAGnostics: Shift-Left Airflow Governance with Policy Enforcement Framework
[URL](https://airflowsummit.org/sessions/2025/dagnostics/)  
Date: 2025-10-09 15:00:00
Summary:
DAGnostics seamlessly integrates Airflow Cluster Policy hooks to enforce governance from local DAG authoring through CI pipelines to production runtime. Learn how it closes validation gaps, collapses feedback loops from hours to seconds, and ensures consistent policies across stages. We examine current runtime-only enforcement and fractured CI checks, then unveil our architecture: a pluggable policy registry via Airflow entry points, local static analysis for pre-commit validation, GitHub Actions CI integration, and runtime hook enforcement. See real-world use cases: alerting standards, resource quotas, naming conventions, and exemption handling. Next, dive into implementation: authoring policies in Python, auto-discovery, cross-environment enforcement, upstream contribution, and testing strategies. We share LinkedIn’s metrics—2,000+ DAG repos, 10,000+ daily executions supporting trunk-based development across isolated teams/use-cases, and 78% fewer runtime violations—and lessons learned scaling policy-as-code at enterprise scale. Leave with a blueprint to adopt DAGnostics and strengthen your Airflow governance while preserving full compatibility with existing systems.


## How Airflow can help with Data Management and Governance
[URL](https://airflowsummit.org/sessions/2025/how-airflow-can-help-with-data-management-and-governance/)  
Date: 2025-10-09 15:00:00
Summary:
Metadata management is a cornerstone of effective data governance, yet it presents unique challenges distinct from traditional data engineering. At scale, efficiently extracting metadata from relational and NoSQL databases demands specialized solutions. To address this, our team has developed custom Airflow operators that scan and extract metadata across various database technologies, orchestrating 100+ production jobs to ensure continuous and reliable metadata collection.
Now, we’re expanding beyond databases to tackle non-traditional data sources such as file repositories and message queues. This shift introduces new complexities, including processing structured and unstructured files, managing schema evolution in streaming data, and maintaining metadata consistency across heterogeneous sources. In this session, we’ll share our approach to building scalable metadata scanners, optimizing performance, and ensuring adaptability across diverse data environments. Attendees will gain insights into designing efficient metadata pipelines, overcoming common pitfalls, and leveraging Airflow to drive metadata governance at scale.


## Modernizing Automation in Secure, Regulated Environments: Lessons from Deploying Airflow
[URL](https://airflowsummit.org/sessions/2025/modernizing-automation-in-secure-regulated-environments/)  
Date: 2025-10-09 15:00:00
Summary:
This session details practical strategies for introducing Apache Airflow in strict, compliance-heavy organizations. Learn how on-premise deployment and hybrid tooling can help modernize legacy workflows when public cloud solutions and container technologies are restricted. Discover how cross-platform engineering teams can collaborate securely using CI/CD bridges, and what it takes to meet rigorous security and governance standards. Key lessons address navigating resistance to change, achieving production sign-off, and avoiding common compliance pitfalls, relevant to anyone automating in public sector settings.


## Building a Transparent Data Workflow with Airflow and Data Catalog
[URL](https://airflowsummit.org/sessions/2025/building-a-transparent-data-workflow-with-airflow-and-data-catalog/)  
Date: 2025-10-09 15:45:00
Summary:
As modern data ecosystems grow in complexity, ensuring transparency, discoverability, and governance in data workflows becomes critical. Apache Airflow, a powerful workflow orchestration tool, enables data engineers to build scalable pipelines, but without proper visibility into data lineage, ownership, and quality, teams risk operating in a black box.
In this talk, we will explore how integrating Airflow with a data catalog can bring clarity and transparency to data workflows. We’ll discuss how metadata-driven orchestration enhances data governance, enables lineage tracking, and improves collaboration across teams. Through real-world use cases, we will demonstrate how Airflow can automate metadata collection, update data catalogs dynamically, and ensure data quality at every stage of the pipeline.


## Enabling SQL testing in Airflow workflows using Pydantic types
[URL](https://airflowsummit.org/sessions/2025/enabling-sql-testing-in-airflow-workflows-using-pydantic-types/)  
Date: 2025-10-09 15:45:00
Summary:
This session explores how to bring unit testing to SQL pipelines using Airflow. I’ll walk through the development of a SQL testing library that allows isolated testing of SQL logic by injecting mock data into base tables. To support this, we built a type system for AWS Glue tables using Pydantic, enabling schema validation and mock data generation. Over time, this type system also powered production data quality checks via a custom Airflow operator. Learn how this approach improves reliability, accelerates development, and scales testing across data workflows.


## Fine-Tuning Airflow: Parameters You May Not Know About
[URL](https://airflowsummit.org/sessions/2025/fine-tuning-airflow/)  
Date: 2025-10-09 15:45:00
Summary:
The Bloomberg Data Platform Engineering team is responsible for managing, storing, and providing access to business and financial data used by financial professionals across the global capital markets. Our team utilizes Apache Airflow to orchestrate data workflows across various applications and Bloomberg Terminal functions. Over the years, we have fine-tuned our Airflow cluster to handle more than 1,000 ingestion DAGs, which has presented unique scalability challenges. In this session, we will share insights into several key Airflow parameters — some of which you may not be all that familiar with — that our team uses to optimize and scale the platform effectively.


## From Repetition to Refactor: Smarter DAG Design in Airflow 3
[URL](https://airflowsummit.org/sessions/2025/from-repetition-to-refactor/)  
Date: 2025-10-09 15:45:00
Summary:
We will explore how Apache Airflow 3 unlocks new possibilities for smarter, more flexible DAG design. We’ll start by breaking down common anti-patterns in early DAG implementations, such as hardcoded operators, duplicated task logic, and rigid sequencing, that lead to brittle, unscalable workflows. From there, we’ll show how refactoring with the D.R.Y. (Don’t Repeat Yourself) principle, using techniques like task factories, parameterization, dynamic task mapping, and modular DAG construction, transforms these workflows into clean, reusable patterns.


## Lessons from Airflow gone wrong: How to set yourself up to scale successfully
[URL](https://airflowsummit.org/sessions/2025/lessons-from-airflow-gone-wrong/)  
Date: 2025-10-09 16:15:00
Summary:
Ever seen a DAG go rogue and deploy itself? Or try to time travel back to 1999? Join us for a light-hearted yet painfully relatable look at how not to scale your Airflow deployment to avoid chaos and debugging nightmares.
We’ll cover the classics: hardcoded secrets, unbounded retries (hello, immortal task!), and the infamous spaghetti DAG where 200 tasks are lovingly connected by hand and no one dares open the Airflow UI anymore. If you’ve ever used datetime.now() in your DAG definition and watched your backfills implode, this talk is for you.


## Model Context Protocol with Airflow
[URL](https://airflowsummit.org/sessions/2025/model-context-protocol-with-airflow/)  
Date: 2025-10-09 16:15:00
Summary:
In today’s data-driven world, effective workflow management and AI are crucial for success. However, there’s a notable gap between Airflow and AI. Our presentation offers a solution to close this gap.
Proposing MCP (Model Context Protocol) server to act as a bridge. We’ll dive into two paths:
AI-Augmented Airflow: Enhancing Airflow with AI to improve error handling, automate DAG generation, proactively detect issues, and optimize resource use. Airflow-Powered AI: Utilizing Airflow’s reliability to empower LLMs in executing complex tasks, orchestrating AI agents, and supporting decision-making with real-time data. Key takeaways:


## Multi-Instance Asset Synchronization - push or pull?
[URL](https://airflowsummit.org/sessions/2025/multi-instance-asset-synchronization-push-or-pull/)  
Date: 2025-10-09 16:15:00
Summary:
As Data Engineers, our jobs regularly include scheduling or scaling workflows.
But have you ever asked yourself, can I scale my scheduling ?
It turns out that you can! But doing so raises a number of issues that need to be addressed.
In this talk we’ll be:
Recapping Asset-aware scheduling in Apache Airflow Discussing diverse methods to upscale our scheduling Solving the issue of maintaining our Airflow Asset synchronized between instances Comparing our professional push based solution and the built-in solution from AIP-82 and the pros and cons of each method. I hope you will enjoy it!


## No More Missed Beats: How Airflow Rescued Our Analytics Pipeline
[URL](https://airflowsummit.org/sessions/2025/no-more-missed-beats-how-airflow-rescued-our-analytics-pipeline/)  
Date: 2025-10-09 16:15:00
Summary:
Before Airflow, our BigQuery pipelines at Create Music Group operated like musicians without a conductor—each playing on its own schedule, regardless of whether upstream data was ready. As our data platform grew, this chaos led to spiralling costs, performance bottlenecks, and became utterly unsustainable.
This talk tells the story of how Create Music Group brought harmony to its data workflows by adopting Apache Airflow and the Medallion architecture, ultimately slashing our data processing costs by 50%. We’ll show how moving to event-driven scheduling with datasets helped eliminate stale data issues, dramatically improved performance, and unlocked faster iteration across teams. Discover how we replaced repetitive SQL with standardized dimension/fact tables, empowering analysts in a safer sandbox.


## Lessons learned for building open source Airflow operators at AWS
[URL](https://airflowsummit.org/sessions/2025/lessons-learned-for-building-open-source-airflow-operators-at-aws/)  
Date: 2025-10-09 16:45:00
Summary:
In this talk, we’ll share our journey and lessons learned from developing a new open-source Airflow operator that integrates a newly-launched AWS service with the Airflow ecosystem. This real-world case study will illuminate the complete lifecycle of building an Airflow operator, from initial design to successful community contribution.
We’ll dive deep into the practical challenges and solutions encountered throughout the journey, including:
Evaluating when to build a new operator versus extending existing ones Navigating the Apache Airflow Open-source contribution process Best practices for operator design and implementation Key learnings and common pitfalls to avoid during the testing and release process Whether you’re looking to contribute to Apache Airflow or build custom operators, this session will provide valuable insights into the development process, common pitfalls to avoid, and best practices when contributing to and collaborating with the Apache Airflow community.


## Lightning talk: Supercharging Apache Airflow: Enhancing Core Components with Rust
[URL](https://airflowsummit.org/sessions/2025/supercharging-apache-airflow-enhancing-core-components-with-rust/)  
Date: 2025-10-09 17:30:00
Summary:
Apache Airflow is a powerful workflow orchestrator, but as workloads grow, its Python-based components can become performance bottlenecks. This talk explores how Rust, with its speed, safety, and concurrency advantages, can enhance Airflow’s core components (e.g, scheduler, DAG processor, etc). We’ll dive into the motivations behind using Rust, architectural trade-offs, and the challenges of bridging the gap between Python and Rust. A proof-of-concept showcasing an Airflow scheduler rewritten in Rust will demonstrate the potential benefits of this approach.


## [Online Reconnect] Introducing Apache Airflow® 3 – The Next Evolution in Orchestration
[URL](https://airflowsummit.org/sessions/2025/airflow-3-online/)  
Date: 2025-10-15 14:00:00
Summary:
In this keynote, core Airflow contributors will walk through key enhancements that boost flexibility, efficiency, and user experience.

## Securing Airflow CLI with API
[URL](https://airflowsummit.org/sessions/2025/securing-airflow-cli-with-api/)  
Date: 2025-10-15 16:00:00
Summary:
This talk will explore the key changes introduced by AIP-81, focusing on security enhancements and user experience improvements across the entire software development lifecycle.

## Cloud Composer - update on what's new!
[URL](https://airflowsummit.org/sessions/2025/google-online/)  
Date: 2025-10-15 16:30:00
Summary:
During this sessions audience is going to learn about newest feature of managed Airflow offering provided by Google Cloud.
If you would like to operate Airflow at scale or in regulated environments then this session is for you.


## Orchestrator of Orchestrators: Uniting Airflow Pipelines with Business Applications in Production
[URL](https://airflowsummit.org/sessions/2025/orchestrator-of-orchestrators-bmc/)  
Date: 2025-10-15 17:00:00
Summary:
Airflow powers thousands of data and ML pipelines—but in the enterprise, these pipelines often need to interact with business-critical systems like ERPs, CRMs, and core banking platforms.
In this demo-driven session we will connect Airflow with Control-M from BMC and showcase how Airflow can participate in end-to-end workflows that span not just data platforms but also transactional business applications.
Session highlights
Trigger Airflow DAGs based on business events (e.g., invoice approvals, trade settlements) Feed Airflow pipeline outputs into ERP systems (e.g., SAP) or CRMs (e.g., Salesforce) Orchestrate multi-platform workflows from cloud to mainframe with SLA enforcement, dependency management, and centralized control. Provide unified monitoring and auditing across data and application layers 

## Using Airflow for Real-Time Data Processing at Scale: Architecture, Challenges & Wins
[URL](https://airflowsummit.org/sessions/2025/airflow-real-time/)  
Date: 2025-10-15 17:30:00
Summary:
Airflow is a powerhouse for batch data pipelines—but can it be tuned for real-time workloads? In this session, we’ll share how we adapted Apache Airflow to orchestrate near-real-time data processing at scale. From leveraging event-driven triggers and external APIs to minimizing latency with smart DAG design, we’ll dive into real-world architectural patterns, challenges, and optimizations that helped us handle time-sensitive data workflows with confidence. This talk is ideal for teams seeking to expand beyond batch and explore hybrid or real-time orchestration using Airflow.


## Simplifying DAG creation with an AI-powered IDE for Airflow
[URL](https://airflowsummit.org/sessions/2025/simplifying-dag-creation-online/)  
Date: 2025-10-15 18:00:00
Summary:
As the demand for data products grows, data engineering teams face mounting pressure to deliver more and even faster, often becoming bottlenecks. Astro IDE changes the game.
Astro IDE is an AI-powered code editor built for Apache Airflow. It helps data teams go from idea to production in minutes—generating production-ready DAGs, enabling in-browser testing, and integrating directly with Git.
In this session, see how Astro IDE accelerates DAG creation, debugging, and deployment so data engineering teams can deliver more, 10x faster.


## Automating Healthcare Triage with Airflow and Large Language Models
[URL](https://airflowsummit.org/sessions/2025/automating-healthcare/)  
Date: 2025-10-15 18:30:00
Summary:
I will talk about how Apache Airflow is used in the healthcare sector with the integration of LLMs to enhance efficiency.
Healthcare generates vast volumes of unstructured data daily, from clinical notes and patient intake forms to chatbot conversations and telehealth reports. Medical teams struggle to keep up, leading to delays in triage and missed critical symptoms. This session explores how Apache Airflow can be the backbone of an automated healthcare triage system powered by Large Language Models (LLMs).


## Managed Workflow for Apache Airflow (MWAA): What's New?
[URL](https://airflowsummit.org/sessions/2025/aws-online/)  
Date: 2025-10-15 19:00:00
Summary:
MWAA is an AWS-managed service that simplifies the deployment and maintenance of the open-source Apache Airflow data orchestration platform. MWAA has recently introduced several new features to enhance the experience for data engineering teams. Features such as Graceful Worker Replacement Strategy that enable seamless MWAA environment updates with zero downtime, IPv6 support, and in place minor Airflow Version Downgrade are some of the many new improvements MWAA has brought to their users in 2025. Last, but not the least, the release of Airflow 3.0 support brings the latest open-source features introducing a new web-server UI, better isolation and security for environments. These enhancements demonstrate Amazon’s continued investment in making Airflow more accessible and scalable for enterprises through the MWAA service.


## Empowering Precision Healthcare with Apache Airflow-iKang Healthcare Group’s DataHub Journey
[URL](https://airflowsummit.org/sessions/2025/empowering-precision-healthcare-with-apache-airflow-ikang-healthcare-groups-datahub-journey/)  
Date: 2025-10-16 14:00:00
Summary:
iKang Healthcare Group, serving nearly 10 million patients annually, built a centralized healthcare data hub powered by Apache Airflow to support its large-scale, real-time clinical operations. The platform integrates batch and streaming data in a lakehouse architecture, orchestrating complex workflows from data ingestion (HL7/FHIR) to clinical decision support.
Healthcare data’s inherent complexity—spanning structured lab results to unstructured clinical notes—requires dynamic, reliable orchestration. iKang uses Airflow’s DAGs, extensibility, and workflow-as-code capabilities to address challenges like multi-system coordination, semantic data linking, and fault-tolerant automation.


## Vayu: The Airflow Copilot
[URL](https://airflowsummit.org/sessions/2025/vayu-airflow-copilot/)  
Date: 2025-10-16 14:30:00
Summary:
Vayu is a conversational copilot for Apache Airflow, developed at Prevalent AI to help data engineers manage, troubleshoot, and fix pipelines using natural language. Deployments often fail silently due to misconfigurations, missing connections, or runtime issues impossible to identify in unit tests. Vayu tackles these via a troubleshooting agent that inspects logs, metrics, configs, and runtime state to find root causes and suggest fixes saving engineers significant troubleshooting time. It can also apply approved fixes to DAG code and commit them to your version control system.


## From Complexity to Simplicity with TaskHarbor: Trendyol's Path to a Unified Orchestration Platform
[URL](https://airflowsummit.org/sessions/2025/from-complexity-to-simplicity-with-taskharbor-trendyol-s-path-to-a-unified-orchestration-platform/)  
Date: 2025-10-16 15:00:00
Summary:
At Trendyol, Turkey’s leading e-commerce company, Apache Airflow powers our task orchestration, handling DAGs with 500+ tasks, complex interdependencies, and diverse environments. Managing on-prem Airflow instances posed challenges in scalability, maintenance, and deployment. To address these, we built TaskHarbor, a fully managed orchestration platform with a hybrid architecture—combining Airflow on GKE with on-prem resources for optimal performance and efficiency.
This talk covers how we:
Enabled seamless DAG synchronization across environments using GCS Fuse. Optimized workload distribution via GCP’s HTTPS & TCP Load Balancers. Automated infrastructure provisioning (GKE, CloudSQL, Kubernetes) using Terraform. Simplified Airflow deployments by replacing Helm YAML files with a custom templating tool, reducing configurations to 10-15 lines. Built a fully automated deployment pipeline, ensuring zero developer intervention. We enhanced efficiency, reliability, and automation in hybrid orchestration by embracing a scalable, maintainable, and cloud-native strategy. Attendees will obtain practical insights into architecting Airflow at scale and optimizing deployments.


## Building an MLOps Platform for 300+ ML/DS Specialists on Top of Airflow
[URL](https://airflowsummit.org/sessions/2025/building-an-mlops-platform-for-300-ml-ds-specialists-on-top-of-airflow/)  
Date: 2025-10-16 16:00:00
Summary:
As your organization scales to 20+ data science teams and 300+ DS/ML/DE engineers, you face a critical challenge: how to build a secure, reliable, and scalable orchestration layer that supports both fast experimentation and stable production workflows. We chose Airflow — and didn’t regret it! But to make it truly work at our scale, we had to rethink its architecture from the ground up.
In this talk, we’ll share how we turned Airflow into a powerful MLOps platform through its core capability: running pipelines across multiple K8s GPU clusters from a single UI (!) using per-cluster worker pools. To support ease of use, we developed MLTool — our own library for fast and standardized DAG development, integrated Vault for secure secret management across teams, enabled real-time logging with S3 persistence and built a custom SparkSubmitOperator for Kerberos-authenticated Spark/Hadoop jobs in Kubernetes. We also streamlined the developer experience — users can generate a GitLab repo and deploy a versioned pipeline to prod in under 10 minutes!


## Driving Analytics with Open Source: Airbyte, dbt, Airflow & Metabase
[URL](https://airflowsummit.org/sessions/2025/driving-analytics/)  
Date: 2025-10-16 17:00:00
Summary:
In this talk, I’ll walk through how we built an end-to-end analytics pipeline using open-source tools ( Airbyte, dbt, Airflow, and Metabase). At WirePick, we extract data from multiple sources using Airbyte OSS into PostgreSQL, transform it into business-specific data marts with dbt, and automate the entire workflow using Airflow. Our Metabase dashboards provide real-time insights, and we integrate Slack notifications to alert stakeholders when key business metrics change.
This session will cover:


## Automating Threat Intelligence with Airflow, XDR, and LLMs using the MITRE ATT&CK Framework
[URL](https://airflowsummit.org/sessions/2025/automating-threat-intelligence-with-airflow-xdr-and-llms-using-the-mitre-att-ck-framework/)  
Date: 2025-10-16 17:30:00
Summary:
Security teams often face alert fatigue from massive volumes of raw log data. This session demonstrates how to combine Apache Airflow, XDR, and LLMs to build automated pipelines for smarter threat triage—grounded in the MITRE ATT&CK framework.
We’ll explore how Airflow can orchestrate a full workflow: ingesting XDR alerts, using LLMs to summarize log events, matching behavior to ATT&CK tactics and techniques, and generating enriched incident summaries. With AI-powered interpretation layered on top of structured threat intelligence, teams can reduce manual effort while increasing context and clarity.


## (Online) Becoming an Apache Airflow Committer from 0
[URL](https://airflowsummit.org/sessions/2025/becoming-an-apache-airflow-committer-from-0-online/)  
Date: 2025-10-16 18:00:00
Summary:
How a Complete Beginner in Data Engineering / Junior Computer Science Student Became an Apache Airflow Committer in Just 5 Months—With 70+ PRs and 300 Hours of Contributions
This talk is aimed at those who are still hesitant about contributing to Apache Airflow. I hope to inspire and encourage anyone to take the first step and start their journey in open-source—let’s build together!


## Dag Versioning in Airflow: Version Proliferation and Open Questions
[URL](https://airflowsummit.org/sessions/2026/dag-versioning-in-airflow-version-proliferation-and-open-questions/)  
Date: 2026-08-31 10:00:00
Summary:
This session explores the next phase of Dag versioning in Airflow and the practical questions users face in real deployments. Dag versioning moved Airflow beyond a “latest only” model, but it also introduced confusion around why Dag versions keep increasing, what disabling Dag bundle versioning actually does, what creates a new version, and how users should think about clears, reruns, and backfills after a Dag changes. I will examine a common misconception: disabling bundle versioning does not stop Dag version changes. I will also connect Dag versioning to Dag delivery in Airflow 3, showing how Git backed Dag bundles provide a more native alternative to git-sync in Helm-based deployments.


## From Airflow 2 to Airflow 3: Migrating 100+ DAGs Without Downtime or Developer Burden
[URL](https://airflowsummit.org/sessions/2026/from-airflow-2-to-airflow-3-migrating-100-dags-without-downtime-or-developer-burden/)  
Date: 2026-08-31 10:00:00
Summary:
Migrating a production Airflow deployment from version 2 to 3 without disrupting hundreds of DAGs across multiple teams sounds scary (and it is). In this talk I will share how we migrated versions without a big-bang cutover, without weeks of cross-team change requests, and without leaving our pipelines in a broken state.
I’ll walk through how we built a compatibility layer to make sure our code runs on both versions during the migration, how we used AI-tooling to orchestrate 400+ DAG changes and how our on-demand ephemeral environments - full k8s deployments deployed for each pull request - helped us experiment and test all the required changes.


## Spec-Driven Development for Airflow DAGs
[URL](https://airflowsummit.org/sessions/2026/spec-driven-development-for-airflow-dags/)  
Date: 2026-08-31 10:00:00
Summary:
AI coding assistants have transformed software development, moving from ad hoc “vibe coding” to rigorous spec-driven development (SDD). The Airflow ecosystem has fully embraced these advancements, but different use cases demand different SDD approaches. This talk compares ETL and ML pipeline patterns, showing how each leverages Airflow’s unique capabilities differently. I then present SDD strategies along a Spec Stability Spectrum. ETL specs are stable and external — schemas, dbt models — making deterministic, template-driven approaches like DAG Factory and the cosmos-dbt-core skill the right fit. ML specs are volatile and internal, as experiments evolve, so LLM-driven hybrid approaches like the Airflow AI SDK and the airflow-hitl skill are better suited. Both approaches are demonstrated live with Claude Code. Examples draw from my work at TXI Digital generating ETL and ML pipelines for heavy industry clients, with a focus on Rail and anecdotes from Renewable Energy.


## Streamlining Data Pipelines Creation at Stripe with Airflow
[URL](https://airflowsummit.org/sessions/2026/streamlining-data-pipelines-creation-at-stripe-with-airflow/)  
Date: 2026-08-31 10:00:00
Summary:
At Stripe, we process petabytes of data daily across thousands of pipelines powering financial reporting, fraud detection, and merchant analytics. As our data estate grew, so did the complexity of authoring, scheduling, and operating these pipelines. Engineers spent more time wrangling Airflow DAG boilerplate and managing dependencies than writing transformation logic.
To address this, we built a declarative platform that generates Airflow DAGs from YAML and SQL definitions. Authors specify what they want — source tables, SQL transformations, incremental mode, output schema — and the platform handles the rest: generating Airflow tasks, wiring upstream sensors, registering Iceberg tables, and configuring scheduling parameters. A key piece is an in-house dataset-to-task mapping service that resolves upstream dataset dependencies to their producing Airflow tasks. When an author declares an input dataset, the platform automatically looks up which task produces it and generates the appropriate sensor — no manual DAG cross-referencing required. This eliminates an entire class of misconfigured dependency bugs common in hand-wired Airflow deployments.


## Airflow to the rescue: managing chemical emergencies
[URL](https://airflowsummit.org/sessions/2026/airflow-to-the-rescue-managing-chemical-emergencies/)  
Date: 2026-08-31 10:30:00
Summary:
At Meteosim, Airflow is the engine for our entire decision system. It runs daily weather and air quality forecasts on schedule, but it also enables OnaChem React, a software that lets users manage chemical emergencies in real-time, and helps us manage consultancy projects.
This talk covers how we set up Airflow 3 to handle five very different types of workloads:
1. Daily Forecasts: Running physics simulations for weather and air quality. 2. Sensor Validation: Ingest data from thousands of sensors and validate it. 3. Human-in-the-Loop: Managing long-running consultancy projects where Dags pause and wait for expert approval. 4. Emergency Response: Help users manage chemical emergencies using multiple real-time toxic dispersion simulations with pre-defined workflows through our SaaS platform. 5. Training AI models: Track multiple experiments. We will explain why Airflow 3 was necessary to make this work. You will see how we orchestrate physics, AI, and human decisions in a single environment.


## Building a low-cost, scalable Airflow Platform for Small Teams
[URL](https://airflowsummit.org/sessions/2026/building-a-low-cost-scalable-airflow-platform-for-small-teams/)  
Date: 2026-08-31 10:30:00
Summary:
Apache Airflow is often perceived as a platform best suited for large organisations with significant infrastructure budgets and dedicated platform teams. In this talk, I want to share how we built and scaled a robust Airflow platform with tight cost constraints whilst still maintaining reliability, governance and developer productivity.
Starting from a small Airflow setup, we have evolved our architecture to support multiple teams and increasingly complex workflows. This includes standardising environments and making sure best practises are adopted around observability, resource management and version control.


## Building storage analytics pipelines for cloud cost optimization with Airflow
[URL](https://airflowsummit.org/sessions/2026/building-storage-analytics-pipelines-for-cloud-cost-optimization-with-airflow/)  
Date: 2026-08-31 10:30:00
Summary:
Storage usage is a major driver of infrastructure cost for media collaboration platforms. Understanding how storage grows across accounts, assets, and workflows requires analytics pipelines that combine product data with infrastructure metrics.
In this talk, I’ll share how we built storage analytics pipelines that model storage usage across accounts and plan tiers to help leadership understand infrastructure cost drivers. Using warehouse data models orchestrated with Airflow, we developed pipelines that track storage usage over time, identify discrepancies in legacy storage calculations, and resolve edge-cases.


## Migrating Airflow at Scale - What the Docs Don't Tell You
[URL](https://airflowsummit.org/sessions/2026/migrating-airflow-at-scale-what-the-docs-don-t-tell-you/)  
Date: 2026-08-31 10:30:00
Summary:
If you are migrating from self-hosted Airflow to any of the managed platforms, most migration guides you’ll find online assume one environment, one team, one version. Large organizations are never that simple.
This talk comes from four years of assisting customers with real migrations across some of the biggest Airflow deployments out there, from self-hosted open source to managed cloud platforms like MWAA, GCC, and Astro, and between major version upgrades.


## Architecting the Center of Excellence: A Strategic Blueprint for Federated Airflow at Scale
[URL](https://airflowsummit.org/sessions/2026/architecting-the-center-of-excellence-a-strategic-blueprint-for-federated-airflow-at-scale/)  
Date: 2026-08-31 11:30:00
Summary:
As Airflow becomes mission-critical, centralized data teams often become a bottleneck. This session provides a framework for building a Center of Excellence (CoE) that empowers autonomous domain teams while maintaining global standards.
We detail the shift toward “Data Platform Engineering,” treating orchestration as a product. Using case studies from large-scale organizations, we discuss a three-layer model: Strategic (governance), Tactical (platform development), and Operational (business unit execution).
Attendees will learn to design a self-service platform with guardrails that manages multiple teams without interference. We will explore using Airflow 3.0’s architecture for task isolation and conclude with a guide on aligning cross-functional teams and measuring value through consumption-based billing.


## DAGs Move Robots: Closed‑Loop Orchestration for Silicon Validation Labs with Airflow
[URL](https://airflowsummit.org/sessions/2026/dags-move-robots-closed-loop-orchestration-for-silicon-validation-labs-with-airflow/)  
Date: 2026-08-31 11:30:00
Summary:
What if your Airflow DAG could orchestrate robots, thermal chambers, and silicon tests, not just code?
Silicon validation labs rely on scarce, stateful physical resources: robotic handlers, DUT boards, thermal/power systems, instruments, and shared hardware queues. Teams often coordinate these via spreadsheets and ad hoc reservations, causing contention, idle gaps, conflicts, poor observability, and slow triage.
This talk presents a closed-loop orchestration model where Apache Airflow is the control plane for a software-defined validation lab. A central DAG coordinates robotic handling, thermal/power setup, stress and performance runs, and parametric characterization on hosts connected to silicon. It continuously ingests hardware health, measurements, and test outcomes, then feeds results into AI-assisted analysis to choose the next physical action: refine parameters, schedule follow-up experiments, or trigger mitigation.


## Debugging the Undebuggable: Lessons from Real Airflow Incidents
[URL](https://airflowsummit.org/sessions/2026/debugging-the-undebuggable-lessons-from-real-airflow-incidents/)  
Date: 2026-08-31 11:30:00
Summary:
Debugging Airflow failures in production can be harder than building the pipelines themselves. Engineers often encounter issues such as disappearing DAGs, hanging tasks, missing logs, zombie tasks, or sudden performance degradation, often with little visibility into the root cause.
Over the past year, while supporting multiple Airflow deployments and integrations, we investigated several such incidents across different teams and environments. This session shares lessons from these real debugging cases and explains how the issues were diagnosed and resolved.


## Enterprise-Grade Airflow Upgrade: Strategies & Deep Dive
[URL](https://airflowsummit.org/sessions/2026/enterprise-grade-airflow-upgrade-strategies-deep-dive/)  
Date: 2026-08-31 11:30:00
Summary:
Participants will learn actionable strategies tailored to their specific upgrade challenges. Supported by a battle-tested approach and guided exercises.

## Taming the MLOps Zoo: Orchestrating and Monitoring Models with Airflow
[URL](https://airflowsummit.org/sessions/2026/taming-the-mlops-zoo-orchestrating-and-monitoring-models-with-airflow/)  
Date: 2026-08-31 11:30:00
Summary:
Thanks to AI, your data scientists can build models faster than ever. The new bottleneck? Their attention. When your team maintains a zoo of ML models (dbt/SQL scoring models, Python ML on Kubernetes, and point-and-click product UI models) every new species adds feeding schedules, health checks, and habitat needs. The real question becomes: which animals need the zookeeper right now?
At Pendo, we orchestrate 10+ ML models through Airflow, each with its own dbt Cloud feature prep, Kubernetes scoring pods, and downstream monitoring. This talk covers how we keep the zoo running: DAG dependencies across heterogeneous model types, conditional execution for models that only score on certain schedules, and model-specific sub-pipelines that keep each species healthy. Then we’ll demo DS ModelGuard, an agentic monitoring system we built internally that does the morning rounds, tracking API health, output volume, likelihood drift, and feature-level input drift, so your data scientists know which enclosure to check first.


## Airflow Autopilot: The Generate-Verify-Refine Loop That Makes Pipeline Authoring Truly AI-Native
[URL](https://airflowsummit.org/sessions/2026/airflow-autopilot-the-generate-verify-refine-loop-that-makes-pipeline-authoring-truly-ai-native/)  
Date: 2026-08-31 12:00:00
Summary:
Today’s Pipeline authoring is synchronous: writing code, chasing error - every step blocks the engineer until resolved. You can’t step away or parallelize. Airflow Autopilot reimagines this to be AI-native and asynchronous. Describe your pipeline’s intent. The agent takes over - orchestrating two classes of purpose-built tools: tools that generate the DAG code and automate setup, and scorer tools that evaluate it across dimensions: e.g. data discovery, auth, compliance, DAG validation, even end-to-end execution. Every scorer returns a deterministic result and structured, prioritized hints. The agent runs the generate → verify → refine loop — calling scorers, reading hints, fixing code, re-scoring — until every dimension passes. You come back to a PR with DAGs that have been iteratively built, tested, and ready for review. For 10,000+ Airflow users, this shifts the engineer from executor to reviewer: you own the intent and final judgment, the agent owns the execution. Attendees leave with the architecture for an AI-native authoring experience, the principles behind decomposing work into scorer-sized verification units, and what it takes to scale this in production.


## Airflow in a Box: Methodology or Madness?
[URL](https://airflowsummit.org/sessions/2026/airflow-in-a-box-methodology-or-madness/)  
Date: 2026-08-31 12:00:00
Summary:
Airflow testing today is a patchwork: you can validate code and catch obvious breakage early, but many production failures live in the seams—runtime state, persistence, serialization boundaries, API behavior, and the way a real deployment executes work across components. The fast tools are valuable, yet they don’t fully model Airflow as a system. Meanwhile, the default development posture nudges you toward single-process behavior and away from realistic concurrency and state interactions. The result is a familiar trade: quick feedback vs. meaningful confidence. “Airflow in a Box” is a step toward collapsing that trade—making deeper, more production-relevant tests accessible without requiring a full, heavyweight instance for every iteration. In this talk, we’ll discuss methodology, quantify slickness, and share real code!


## Designing Self-Healing Airflow Platforms: Autonomous DAG Recovery at Scale
[URL](https://airflowsummit.org/sessions/2026/designing-self-healing-airflow-platforms-autonomous-dag-recovery-at-scale/)  
Date: 2026-08-31 12:00:00
Summary:
Most Airflow failures are still handled manually — retries, Slack alerts, and late-night debugging. This talk shows how to design Airflow as a self-healing platform that detects problems early, limits blast radius, and automatically recovers. We’ll cover practical patterns for DAG, schema, and dependency-drift detection; safe, selective backfills; predictive failure modeling using metadata; lineage-aware rollbacks; and canary deployment for DAGs. You’ll learn how to isolate unstable workloads before they impact others and how to turn Airflow into an intelligent control plane — not just a scheduler.


## Orchestrating Graph Database Workloads in Apache Airflow with Apache TinkerPop
[URL](https://airflowsummit.org/sessions/2026/orchestrating-graph-database-workloads-in-apache-airflow-with-apache-tinkerpop/)  
Date: 2026-08-31 12:00:00
Summary:
Graph databases are increasingly used for relationship-heavy data such as fraud detection, knowledge graphs and CRM systems, yet integrating them into orchestration workflows has remained difficult. This session introduces the Apache TinkerPop Provider for Airflow, enabling graph databases to be orchestrated as first-class citizens. I will demonstrate how it works with both self-hosted and managed services such as AWS Neptune and Azure Cosmos DB.


## Airflow 3.0 Asset Watchers: Cross-Domain Data Mesh Orchestration with AI-Assisted Deployment
[URL](https://airflowsummit.org/sessions/2026/airflow-3-0-asset-watchers-cross-domain-data-mesh-orchestration-with-ai-assisted-deployment/)  
Date: 2026-08-31 12:30:00
Summary:
Data Mesh decentralises data ownership across business domains. In regulated industries each domain operates in its own account where producers publish data products and consumers subscribe. This enforces governance, limits blast radius and preserves autonomy. When each domain runs its own Airflow, orchestrating across these boundaries is the central challenge. Airflow 2.4 introduced data-aware scheduling which were designed for single Airflow instance with no native cross-instance event propagation. In practice this meant building polling sensors that queried the producer REST API to check upstream completion, but it is unreliable as events were lost and ordering not guaranteed. Airflow 3.0 resolves this with Event-driven scheduling via AssetWatcher. The Triggerer monitors a message queue and triggers the consumer DAG when the producer publishes a completion event. This talk traces that journey through a regulated enterprise Data Mesh. We also share how we built an agentic AI skills framework that encodes operational Airflow knowledge into reusable skills, enabling an AI agent to autonomously deploy, validate and troubleshoot the cross-environment pattern end-to-end.


## Airflow Callbacks Revamped: Beyond the Dag Processor
[URL](https://airflowsummit.org/sessions/2026/airflow-callbacks-revamped-beyond-the-dag-processor/)  
Date: 2026-08-31 12:30:00
Summary:
Airflow’s callback system has undergone significant architectural changes recently. Originally driven by the introduction of Deadline Alerts, these improvements have far broader implications for how callbacks are defined, where they run, and how reliable they are.
In this talk, I’ll cover the user-facing and provider-facing changes along with a brief look at the significant technical design decisions and internal refactoring behind them, such as a new workload type and unified type-agnostic database model for callbacks. In the long term, this work makes both callbacks and the Dag Processor more robust, and the improved isolation is a key stepping stone toward Airflow’s upcoming multi-team capabilities.


## Event-Driven Orchestration Monitoring: Streaming Airflow Metadata to Kafka via CDC
[URL](https://airflowsummit.org/sessions/2026/event-driven-orchestration-monitoring-streaming-airflow-metadata-to-kafka-via-cdc/)  
Date: 2026-08-31 12:30:00
Summary:
How do you monitor Airflow across 50 teams in real-time? How do downstream systems react instantly to pipeline completions without polling APIs? How do you build custom dashboards without overloading Airflow’s database? This talk demonstrates how we use Change Data Capture to stream Airflow’s metadata to Kafka, making orchestration events consumable by any system in real-time. By capturing changes in Airflow’s Postgres database and publishing them to Kafka topics, we enable instant notifications, real-time dashboards, compliance audit trails, and cross-system orchestration without modifying Airflow code or impacting performance. You’ll learn how to set up Debezium CDC for Airflow’s metadata tables, design Kafka topics for task and DAG events, build real-time consumers for monitoring and alerting, handle schema evolution across Airflow upgrades, and implement cost attribution and SLA monitoring in real-time. Using production examples processing millions of events daily, I’ll share architecture decisions, performance optimizations, and lessons from running CDC at scale. You’ll leave with patterns for making Airflow observable to your entire organization.


## Optimising Airflow in Real-World Deployments: Profiling, Performance Drift, and Confident Upgrades
[URL](https://airflowsummit.org/sessions/2026/optimising-airflow-in-real-world-deployments-profiling-performance-drift-and-confident-upgrades/)  
Date: 2026-08-31 12:30:00
Summary:
Performance issues in Apache Airflow rarely appear as clear failures. Instead, they surface as subtle signals: longer task queue times, slower DAG parsing, scheduler lag, or workers hitting limits as workloads grow.
In this talk, we share lessons from profiling real production deployments across Airflow 2.x and 3.x. Combining frontline operational insights with focused technical investigation, we analysed task latency, DAG parsing time, worker behaviour, and metadata database performance under sustained load.


## Anatomy of a Task Instance: From Scheduled to Done
[URL](https://airflowsummit.org/sessions/2026/anatomy-of-a-task-instance-from-scheduled-to-done/)  
Date: 2026-08-31 13:00:00
Summary:
In this session I will provide a deep dive into a task instance’s lifetime. From when the scheduler decides for it to be scheduled until it is marked as success or failed.
We will explore when in the process concepts like concurrency, pools and priority weights apply, what it means for a task to be “queued” and where things like cluster policies, operator links, callbacks and event listeners are evaluated.


## Deadlines for DAGs: What's Shipped and What's Next
[URL](https://airflowsummit.org/sessions/2026/deadlines-for-dags-what-s-shipped-and-what-s-next/)  
Date: 2026-08-31 13:00:00
Summary:
Airflow’s legacy SLA (Service level agreement) feature let users set a maximum expected duration for a DAG run and receive an email when it was exceeded, but it was inflexible and hard to configure. Deadline Alerts replaced it in 3.1 with a general-purpose system for time-based alerting. Since then, two release cycles have reshaped the feature.
Callbacks now run in supervised subprocesses with access to Connections, Variables, and Assets, which means they can query your infrastructure and respond to problems, not just send a notification. Deadline status is visible in the UI Grid view and DAG run overview. Named deadlines let you attach multiple alerts to a single DAG for different stakeholders. OpenLineage captures deadline events. And fixes for duplicate callbacks under HA schedulers and migration performance have made the feature production-solid.


## From Cron to Assets: Event-Driven Drone Telemetry Ingestion with Airflow
[URL](https://airflowsummit.org/sessions/2026/from-cron-to-assets-event-driven-drone-telemetry-ingestion-with-airflow/)  
Date: 2026-08-31 13:00:00
Summary:
A drone doesn’t care what time it is. It takes off when the mission says so, lands when the battery says so, and uploads its logs whenever the LTE link or WiFi finally cooperates. Cron-based pipelines, by contrast, care deeply about the clock — and that mismatch is where most fleet telemetry stacks quietly bleed money, latency, and engineer sanity on empty polls, half-parsed flights, and workers pinned waiting on slow uploads.


## Asset Partitions: Matching Workflow to the Right Data
[URL](https://airflowsummit.org/sessions/2026/asset-partitions-matching-workflow-to-the-right-data/)  
Date: 2026-08-31 14:30:00
Summary:
Asset partitions are a key building block in Expanded Data Awareness. This session explains the core semantics of partition definitions, partition mappings, and backfill behavior in AIP-76. I will show how these pieces fit together in the current design, then discuss where asset partitions can go next, including improvements in authoring ergonomics, observability, and partition-aware workflow capabilities. Attendees will leave with a clear mental model of today’s implementation and a practical view of future direction.


## Build AI Pipelines with Apache Airflow 3
[URL](https://airflowsummit.org/sessions/2026/build-ai-pipelines-with-apache-airflow-3/)  
Date: 2026-08-31 14:30:00
Summary:
Learn how to build an end-to-end AI pipeline using Airflow’s LLM task decorators, all in your browser, no setup required.

## Orchestrating AI-Enabled Prescription Workflows with Apache Airflow: Improving Accuracy, Efficiency,
[URL](https://airflowsummit.org/sessions/2026/orchestrating-ai-enabled-prescription-workflows-with-apache-airflow-improving-accuracy-efficiency/)  
Date: 2026-08-31 14:30:00
Summary:
Modern pharmacy enterprise systems must process high volumes of complex prescriptions while maintaining strict safety, compliance, and operational efficiency. However, traditional rule-based platforms frequently generate low-specificity alerts that contribute to alert fatigue, workflow bottlenecks, and increased manual intervention. As clinical guidelines, payer requirements, and treatment protocols evolve, static rule engines struggle to keep pace with the dynamic nature of modern pharmacy operations.
This session presents a practical architecture for AI-enabled prescription workflow automation orchestrated through Apache Airflow, enabling scalable, transparent, and auditable clinical workflows. By combining rule-based safety checks with machine learning models for classification, anomaly detection, and intelligent workflow routing, the system significantly improves routing precision, reduces false positives, and accelerates prescription verification.


## Orchestrating Streaming Data Pipelines with Airflow, Kafka, Spark, and Kubernetes on GCP
[URL](https://airflowsummit.org/sessions/2026/orchestrating-streaming-data-pipelines-with-airflow-kafka-spark-and-kubernetes-on-gcp/)  
Date: 2026-08-31 14:30:00
Summary:
Modern data platforms rely on real-time pipelines to process and analyze large volumes of streaming events. Apache Airflow is widely used for batch orchestration, but it can also coordinate complex streaming architectures. In this session, we explore how Airflow orchestrates scalable pipelines built with Apache Kafka and Apache Spark running on Kubernetes in cloud environments.
We walk through an architecture where Kafka handles high-throughput event ingestion, Spark processes streaming data for analytics and transformation, and Kubernetes provides scalable infrastructure for distributed workloads. Airflow acts as the orchestration layer, coordinating job scheduling, pipeline dependencies, and operational visibility.


## Streamlining Your Airflow Upgrade: Essential Tools for Migrating from 2.x to 3
[URL](https://airflowsummit.org/sessions/2026/streamlining-your-airflow-upgrade-essential-tools-for-migrating-from-2-x-to-3/)  
Date: 2026-08-31 14:30:00
Summary:
Airflow 3 has officially arrived! If you’re considering an upgrade, this session will equip you with essential migration utilities that facilitate a smooth transition from Airflow 2.x. Attendees will learn the new CLI command, “airflow config lint”, to analyze your configuration files for any removed, deprecated, or renamed elements. This command provides comprehensive feedback and allows for filtering specific sections and options.
During the session, attendees will learn to leverage a set of rigorous Ruff rules - AIR301, AIR302, and AIR303 - crafted to detect migration issues within your codebase automatically. Notably, rule AIR301 flags DAG definitions lacking an explicit schedule argument, a critical update in Airflow 3. Rule AIR302 identifies deprecated functions and removes configuration settings, offering recommended alternatives. Rule AIR303 highlights code that references components now shifted to provider packages, ensuring your integrations are up to date.


## Breaking the Monolith: Implementing Airflow 3.x Remote Execution for Multi-Team Environments
[URL](https://airflowsummit.org/sessions/2026/breaking-the-monolith-implementing-airflow-3-x-remote-execution-for-multi-team-environments/)  
Date: 2026-08-31 15:00:00
Summary:
Problem Statement: As our data platform scaled, our shared Airflow 2.9 deployment became a bottleneck with critical challenges: development friction from shared repositories, custom security workarounds, release coordination complexity, data isolation concerns, and cost attribution opacity. When Airflow 3.x launched with hybrid execution support, we restructured our architecture. Following a successful proof of value, we implemented remote execution - enabling teams to run workloads in isolated Kubernetes clusters while maintaining centralized orchestration. This session shares our journey, architectural decisions, and how we leveraged agentic AI to streamline migration and developer experience.


## Declarative Pipelines Meet Declarative Orchestration: Spark Declarative Pipelines + Airflow 3
[URL](https://airflowsummit.org/sessions/2026/declarative-pipelines-meet-declarative-orchestration-spark-declarative-pipelines-airflow-3/)  
Date: 2026-08-31 15:00:00
Summary:
Apache Spark’s new Declarative Pipelines (SDP) let engineers define WHAT their data should look like, not HOW to build it. Apache Airflow 3 brings a declarized orchestration model. Together, they eliminate an entire category of boilerplate: the DAG that exists only to babysit a pipeline. This talk walks through building a production Spark SDP pipeline orchestrated by Airflow 3, showing how dependency graphs replace imperative task chains, how testing and recovery patterns change when your pipeline is declarative end-to-end, and what this means for the 80% of data engineering time currently spent on operational plumbing.


## Healthcare Interoperability Meets Airflow Extensibility
[URL](https://airflowsummit.org/sessions/2026/healthcare-interoperability-meets-airflow-extensibility/)  
Date: 2026-08-31 15:00:00
Summary:
In healthcare data, standards are often anything but standard. Every new partner arrives with its own requirements for data exchange spanning FHIR APIs, HL7 feeds, SFTP drops, and custom vendor extracts.
The result? Integration projects that stretch from weeks into months, custom pipelines that only one engineer understands, and implementation teams who are already counting down to your next missed deadline.
This session shows how Airflow can change your approach to managing data transfer for healthcare partners.


## Migrating Airflow 2 to 3 for Infrastructure Operations at Scale
[URL](https://airflowsummit.org/sessions/2026/migrating-airflow-2-to-3-for-infrastructure-operations-at-scale/)  
Date: 2026-08-31 15:00:00
Summary:
This talk covers migrating a production Airflow platform that orchestrates a large VM fleet — provisioning, OS patching, and decommissioning at high concurrency. This is not a data pipeline — it is infrastructure operations at fleet scale We’ll share workflow patterns that make fleet-scale orchestration possible in Airflow, then cover how we moved from an Airflow 2 monolith — all components on every node with fixed worker counts — to Airflow 3 with independently scalable services, each with its own release cycle. We’ll dig into a silent breaking change in Airflow 3’s XCom behavior: xcom_pull(key=…) without task_ids no longer searches upstream tasks, returning None with no warning. We’ll present three iterations of solving this — from O(n) DAG traversal to a custom XCom backend that restores Airflow 2 semantics with zero DAG code changes — and the design tradeoffs at each stage. Attendees will learn how Airflow powers infrastructure operations beyond data pipelines, how Airflow 3’s XCom silently breaks Airflow 2 workflows, three approaches to the same migration problem, and lessons from running both versions in parallel.


## Beyond Containers: Securely Orchestrating AI Agents with Strong Isolation in Airflow
[URL](https://airflowsummit.org/sessions/2026/beyond-containers-securely-orchestrating-ai-agents-with-strong-isolation-in-airflow/)  
Date: 2026-08-31 15:30:00
Summary:
AI agents break the traditional Airflow trust model. While standard tasks are deterministic, agents execute dynamic logic and invoke external tools, meaning untrusted code is suddenly running inside standard containers sharing your host kernel. This session demonstrates how to secure AI workloads in Airflow without rewriting the orchestrator or building custom executors. We will introduce a custom, policy-driven @agent TaskFlow abstraction that leverages Kubernetes executor_config overrides (like runtimeClassName) to isolate workloads on the fly.


## Designing Domain-Oriented dbt Projects and Making Them Work in Airflow
[URL](https://airflowsummit.org/sessions/2026/designing-domain-oriented-dbt-projects-and-making-them-work-in-airflow/)  
Date: 2026-08-31 15:30:00
Summary:
As analytics teams grow, monolithic dbt projects can become tightly coupled and difficult to scale. Cross-domain dependencies multiply, deployment cycles slow down, and ownership boundaries blur.
dbt Mesh proposes a domain-oriented approach with independently owned dbt projects, explicit cross-project contracts, and controlled exposure to dependencies. Applying Mesh principles is not just about splitting repositories; orchestration must also support these boundaries.
In this session, we explore how to design dbt projects according to Mesh principles and how Airflow orchestration can reinforce those architectural decisions. Using multi-project capabilities in Cosmos that leverage dbt Loom-style cross-project referencing, we demonstrate how Airflow can model domain separation while still enabling controlled cross-project dependencies.


## Developer Velocity at Scale: Production-Like Airflow Environments on Kubernetes
[URL](https://airflowsummit.org/sessions/2026/developer-velocity-at-scale-production-like-airflow-environments-on-kubernetes/)  
Date: 2026-08-31 15:30:00
Summary:
Teams running Airflow on Kubernetes know the trade‑off all too well: Kubernetes scales beautifully in production, but makes local development slow, brittle, and unrealistic. Engineers struggle to replicate production environments locally, forcing them into inefficient “test-in-production” cycles that slow delivery velocity, increase deployment risk, and frustrate data teams.
In this talk, we’ll walk through the architectural patterns and platform engineering approach we used to give engineers on‑demand, isolated, production‑like Airflow environments, without sacrificing the benefits of shared Kubernetes infrastructure.


## From Hours to Minutes: Orchestrating Local LLMs for Sensitive Data Pipelines with Apache Airflow
[URL](https://airflowsummit.org/sessions/2026/from-hours-to-minutes-orchestrating-local-llms-for-sensitive-data-pipelines-with-apache-airflow/)  
Date: 2026-08-31 15:30:00
Summary:
Processing unstructured data in regulated industries, healthcare, finance, legal, is one of the hardest data engineering challenges: the data is messy, privacy constraints prevent sending it to external APIs, and scale makes manual processing impossible.
In this talk, I’ll walk through how to design and deploy an Apache Airflow–orchestrated LangChain pipeline powered by LLMs to digitize unstructured documents into a unified structured platform.
I’ll cover the full architecture: how Airflow DAGs coordinate multi-step LLM inference, validation, and ingestion stages; how LoRA/PEFT fine-tuning adapted open-source LLMs for domain-specific language without leaking sensitive data; and how failure handling, retries, and data quality checks were built natively into Airflow.


## Building Blocks, not Factories: Abstractions that enable, not obscure
[URL](https://airflowsummit.org/sessions/2026/60.-building-blocks-not-factories/)  
Date: 2026-08-31 16:00:00
Summary:
Many teams develop their own “Dag factory” to make Airflow easier to use in their organizations. This can help their users avoid python and configure Dags in a simpler manner. However, there is a huge spike in the difficulty curve of writing a DAG if it requires logic that does not fit into the confines of the Dag factory. If you want to create such a DAG, you are then having to completely abandon the pre-made framework and go back to writing a pure airflow DAG. Instead, I will present a different perspective that instead of producing entire DAGs, you should create pre-made task groups that can be dropped into a DAG to cover common steps, but in a manner that maintains a smooth difficulty curve if you want to add customer elements.


## Fixing The Token Authentication: Revocation, Scoping, and Securing the Execution Boundary
[URL](https://airflowsummit.org/sessions/2026/fixing-the-token-authentication-revocation-scoping-and-securing-the-execution-boundary/)  
Date: 2026-08-31 16:00:00
Summary:
When Airflow 3 introduced JWT based task authentication, it also introduced new attack surfaces; such as, Tokens that can’t be revoked,Tasks that lose authentication while waiting in queues and Forked processes that inherit signing keys and also can forge tokens for other tasks.
In this talk, I’ll walk through three security challenges at the task execution boundary and the code contributed to fix them:
Token revocation (merged, PR #61339): Airflow 3.x had no way to invalidate issued JWTs with implications for common compliance frameworks.


## Gleaming the Cube: Exploring the limits of Airflow through the Rubik's Cube
[URL](https://airflowsummit.org/sessions/2026/gleaming-the-cube-exploring-the-limits-of-airflow-through-the-rubik-s-cube/)  
Date: 2026-08-31 16:00:00
Summary:
What does solving a Rubik’s Cube have to do with Apache Airflow? More than you’d think.
In this talk, I’ll walk through a project where Airflow orchestrates the process of solving a Rubik’s Cube — not as a gimmick, but as a framework for exploring cyclic workflows, state management, and iterative computation in a system designed for DAGs. Cube-solving algorithms naturally require feedback loops, evolving state, and conditional branching — all things that challenge Airflow’s acyclic model.


## One Gateway, Six Clusters: Routing, Federation, and Zero-Downtime Upgrades at eBay
[URL](https://airflowsummit.org/sessions/2026/one-gateway-six-clusters/)  
Date: 2026-08-31 16:00:00
Summary:
We built a centralized Gateway that sits in front of our entire scheduling fleet and solves three problems no single-cluster Airflow setup ever faces.
Composite Routing — Workflows are bound to clusters via a tag or their workspace
Global Concurrency Control — Each cluster enforces its own Airflow pool locally, unaware of what the other five are running. Shared downstream systems — rate-limited APIs, licensed compute engines — can be overwhelmed even when every individual pool looks healthy. The Gateway acts as a platform-wide slot broker: operators acquire a slot before doing real work. A built-in heartbeat scheduler reconciles stale slots against each cluster’s REST API, handling crashes and OOM kills transparently.


## The State of Airflow: Momentum, Innovation, and What's Next
[URL](https://airflowsummit.org/sessions/2026/the-state-of-airflow/)  
Date: 2026-08-31 17:30:00
Summary:
Airflow 3 has been out for a year. In this keynote, we take stock of where the community stands, what we built together, and where we are headed.
We open with the data: adoption trends, community growth, and honest feedback from teams running Airflow 3 in production. What is working, what surprised us, and what the survey tells us about how the ecosystem is evolving.
The second section covers the year in Airflow. Provider discovery and distribution has been modernized. Airflow gained first-class support for AI and LLM workloads. And scheduling became more powerful, letting pipelines respond to data at a finer granularity.


## Data Engineers Already Solved Agentic AI's Reliability Problem
[URL](https://airflowsummit.org/sessions/2026/data-engineers-already-solved-agentic/)  
Date: 2026-08-31 9:10:00
Summary:
Understand how Airflow 3 can help you solve the “messy middle” pattern so that you don’t need to use a streaming engine.

## Multi-Team Airflow: A Customer-Driven Journey
[URL](https://airflowsummit.org/sessions/2026/multi-team-airflow-a-customer-driven-journey/)  
Date: 2026-08-31 9:10:00
Summary:
Explore building multi-team support in Airflow, working backwards from real deployment challenges and community pain points we’ve observed.

## From Experiments to Production: How We Built a Lightweight ML Platform on Airflow
[URL](https://airflowsummit.org/sessions/2026/from-experiments-to-production-how-we-built-a-lightweight-ml-platform-on-airflow/)  
Date: 2026-08-31 9:30:00
Summary:
We’ll share how we use Airflow to orchestrate production ML end-to-end: from feature pipelines in Snowflake, to model training and artifact promotion, to batch scoring and prediction delivery.

## The Self-Healing Pipeline: How Error Classification Eliminated 100% of Engineering Oversight
[URL](https://airflowsummit.org/sessions/2026/the-self-healing-pipeline-how-error-classification-eliminated-100-of-engineering-oversight/)  
Date: 2026-08-31 9:30:00
Summary:
What if your pipeline could tell the difference between recoverable errors and real bugs and handle both without waking anyone up?

## Airflow at the heart of Equifax's Data Processing
[URL](https://airflowsummit.org/sessions/2026/airflow-at-the-heart-of-equifaxs-data-processing/)  
Date: 2026-09-01 09:00:00
Summary:
At Equifax, Apache Airflow is used across many departments, helping Data Engineers, Data Scientists, and Business Analysts in their daily work.
This presentation is about how to use modern orchestration technology at the heart of data processing and business processes to support daily company operations.


## Airflow as a Platform: Operating High-Volume, Enterprise-Scale Orchestration
[URL](https://airflowsummit.org/sessions/2026/airflow-as-a-platform/)  
Date: 2026-09-01 09:30:00
Summary:
What happens when your Airflow footprint grows from a single team’s scheduler to a fully managed enterprise data platform? You stop thinking about individual DAGs and start thinking about fleet operations, tenant isolation, automated upgrades, and self-service authoring at scale.
In this talk, we share our journey building a multi-tenant orchestration platform on Apache Airflow running hundreds of isolated instances on Kubernetes. We will dive deep into:
The Architecture: How we achieved strict tenant isolation and automated infrastructure provisioning on Kubernetes. Fleet Operations: Our strategy for handling seamless, zero-downtime upgrades across hundreds of instances. The Abstraction Layer: How we built a self-service experience that allows any team—technical or non-technical—to go from idea to production pipeline without managing infrastructure. Whether you are looking to scale your current Airflow deployment or are managing Airflow at an enterprise level, you will walk away with actionable architectural patterns and real-world operational lessons.


## Airflow as a Harness: The Workflow That Merged Itself
[URL](https://airflowsummit.org/sessions/2026/airflow-as-a-harness-the-workflow-that-merged-itself/)  
Date: 2026-09-01 10:30:00
Summary:
This talk is the story of getting a PR merged into a Apache Airflow without writing a single line of code, using Apache Airflow itself as an agentic orchestration harness to replicate the functionality of Claude Code for any pluggable LLM.
We’ll walk through how Airflow’s AIP-99 Dag functionality map naturally onto the tool-use loops, context management, and decision branching that power modern agentic coding workflows. The result is a model-agnostic harness that can read a codebase, reason about changes, write and test code, and deploy a commit to a git repository, all orchestrated as an Airflow Dag.


## Common Issues When Running dbt in Airflow (and How to Fix Them)
[URL](https://airflowsummit.org/sessions/2026/common-issues-when-running-dbt-in-airflow-and-how-to-fix-them/)  
Date: 2026-09-01 10:30:00
Summary:
In many modern data platforms, orchestration tools are combined with transformation frameworks. A common pattern is orchestrating dbt (data build tool) transformations using Apache Airflow — something reported by roughly 44% of the community.
At first glance, the integration seems straightforward: simply run dbt run inside an Airflow task. Some teams go further and use libraries that convert dbt projects into native Airflow DAGs, such as Astronomer Cosmos.
In practice, however, teams quickly run into operational and architectural challenges. Slowness, out-of-memory errors, zombie tasks, and DAGs that take minutes to appear in the UI are just a few of the issues that can emerge as projects scale.


## From Chaos to Control: Navigating Airflow Sprawl with Centralized Observability
[URL](https://airflowsummit.org/sessions/2026/from-chaos-to-control-navigating-airflow-sprawl-with-centralized-observability/)  
Date: 2026-09-01 10:30:00
Summary:
As data platforms mature, organizations often experience “Airflow Sprawl”—the rapid, organic growth of isolated Airflow instances across different teams and projects. While this empowers localized control, it creates dangerous silos that hinder visibility, increase operational risk, and erode developer productivity. In this session, we will explore the critical challenges of managing a fragmented Airflow ecosystem and discuss strategies for regaining control. We will examine why centralizing execution history and establishing unified observability is essential for reducing Mean Time to Recovery (MTTR), mitigating hidden security risks, and transforming fragmented instances into a cohesive, reliable data service. Attendees will leave with a strategic framework for managing Airflow at scale.


## Stop Fighting Fires: Autonomous Incident Response for Apache Airflow using AI-Powered Agents
[URL](https://airflowsummit.org/sessions/2026/stop-fighting-fires-autonomous-incident-response-for-apache-airflow-using-ai-powered-agents/)  
Date: 2026-09-01 10:30:00
Summary:
Explore how an AI-powered DevOps agent can supercharge Airflow operations, from automated DAG failure diagnosis and intelligent log analysis to proactive prevention of recurring incidents.

## Using agents to create ML pipelines with DAG factory on MWAA serverless
[URL](https://airflowsummit.org/sessions/2026/using-agents-to-create-ml/)  
Date: 2026-09-01 10:30:00
Summary:
Get hands-on experience building an end-to-end ML pipeline on Amazon Managed Workflows for Apache Airflow (MWAA) Serverless.

## 850 Engineering Hours Back, Every Month: Context-Aware Agentic Airflow Platform at Wix
[URL](https://airflowsummit.org/sessions/2026/850-engineering-hours-back-every-month/)  
Date: 2026-09-01 11:00:00
Summary:
Generic AI coding assistants like Cursor and Claude Code are powerful, but they struggle with proprietary infrastructures. At Wix, managing 8000+ active DAGs across 110 Data Engineers, we found that standard AI tools lacked the context to be truly effective - they didn’t know our custom operators, DWH modeling patterns, or strict governance rules. In this session, we’ll show how we built a context-aware agentic coding platform that bridges this gap across the full pipeline lifecycle. You will learn how we enabled our coding agents to: - Write compliant DAGs using agent rules, skills, and plugins that encode production standards, naming conventions, and modeling patterns - distributed across the organization via our internal AICM tool - Debug locally using skills that tap directly into running Airflow containers, and surface logs and fix failures without leaving your IDE - Resolve production incidents autonomously via AirBot, our Slack-native AI agent that catches pipeline failures, pulls the DAG’s logs, and suggests fixes before the on-call even sees the alert Whether you’re trying to optimize your team’s workflows or curious how far coding agents can go today, join us in this exciting talk.


## Developing an AI-powered personal endurance sports coach
[URL](https://airflowsummit.org/sessions/2026/developing-an-ai-powered-personal-endurance-sports-coach/)  
Date: 2026-09-01 11:00:00
Summary:
During this session, I’ll deep dive into the implementation of an AI-powered endurance sports coach using Apache Airflow as the backbone for data ingestion and processing. Beyond data pipelines, I’ll explain what’s required to build a conversational AI system, from structured data modeling to orchestration and retrieval. We’ll explore how metrics are precomputed, how vector search enables contextual memory, and which front-end patterns work best for interacting with AI agents. The result is a reproducible architecture where Airflow powers the data layer and an LLM provides the reasoning on top to help athletes perform at their best in numbers-driven endurance sports.


## Stop Debugging in Production: Shift-Left Airflow for Streaming Pipelines with AI
[URL](https://airflowsummit.org/sessions/2026/stop-debugging-in-production/)  
Date: 2026-09-01 11:00:00
Summary:
As Apache Airflow expands beyond batch into real-time, event-driven architectures, data teams face a new set of challenges: duplicated DAG patterns, fragile Kafka-triggered workflows, and debugging cycles that happen too late—often in production.
In this session, we introduce a shift-left approach to pipeline reliability for environments combining Airflow with streaming platforms like Confluent. We’ll explore how event-driven pipelines increase complexity—and why traditional debugging and validation approaches no longer scale. You’ll see how IBM Bob, an AI-powered assistant for data engineers, brings real-time code review, refactoring guidance, and debugging insights directly into developer workflows.


## Agentic Pipelines on Airflow: From Thesis to Production
[URL](https://airflowsummit.org/sessions/2026/agentic-pipelines-on-airflow-from-thesis-to-production/)  
Date: 2026-09-01 11:30:00
Summary:
The industry treats agents and pipelines as opposing paradigms. We think that framing is wrong. Most agentic problem-solving, when you look at what it actually does, has pipeline structure: gather data, process each dimension independently, synthesize, evaluate. The question is not “agents or pipelines?” but where the LLM fits inside the pipeline and what you gain by making each step explicit.
This talk makes that concrete. We start with AIP-99 and the operator library that gives Airflow first-class LLM support: inference, SQL generation, branching, schema validation, and embedding, all backed by PydanticAI with 20+ model providers out of the box. We walk through a real pipeline that analyzes 5,856 survey responses using four parallel LLM-generated queries, DataFusion execution, and a synthesis step, showing exactly where the LLM reasons and where the pipeline handles everything else.


## From Data Pipelines to Business Outcomes: Unifying Airflow and Enterprise Workflows
[URL](https://airflowsummit.org/sessions/2026/from-data-pipelines-to-business-outcomes/)  
Date: 2026-09-01 11:30:00
Summary:
Airflow has become a foundational platform for data, analytics, and AI pipelines. But business outcomes often depend on workflows that extend beyond the data platform to ERP systems, CRM applications, cloud services, and operational processes.
In this demo-driven session, we’ll show how organizations connect Airflow with broader enterprise workflows to create end-to-end automation across technologies and teams. Learn how enterprises are moving beyond isolated pipelines toward a unified orchestration strategy that delivers business outcomes.


## Spark + Airflow: How Orchestration Decisions Impact Performance and Cost
[URL](https://airflowsummit.org/sessions/2026/spark-airflow-how-orchestration-decisions-impact-performance-and-cost/)  
Date: 2026-09-01 11:30:00
Summary:
Explore key design questions when orchestrating Spark with Airflow. Using real production scenarios, we’ll examine the tradeoffs between orchestration strategies.

## How We Taught an Agent Airflow: Building Self-Healing Pipelines with Otto
[URL](https://airflowsummit.org/sessions/2026/how-we-taught-an-agent-airflow/)  
Date: 2026-09-01 12:00:00
Summary:
Generic AI assistants can read your logs, but that doesn’t make them good at diagnosing your pipelines. When a Dag fails, the evidence is scattered across task logs, lineage, code diffs, and upstream data, and Airflow is the only place it all comes together. This session is about what happens when you build an investigation agent there, instead of bolting one onto your logs.
We’ll show how Otto, Astronomer’s data engineering agent, encodes failure patterns from eight years of running Airflow at enterprise scale and grounds every investigation in the live context of your own environment. We’ll share real examples of teams using Otto in production, including one that cut MTTR by 95% in eight weeks and now starts their mornings reviewing proposed fixes instead of firefighting.


## Advanced Deadline Alerts: Writing your own custom References and Callbacks
[URL](https://airflowsummit.org/sessions/2026/advanced-deadine-alerts-writing-your-own-custom-references-and-callbacks/)  
Date: 2026-09-01 13:30:00
Summary:
Learn how to build your own deadline alerts on DAGs and fire callbacks when deadlines are missed.

## Orchestrating AI-Ready Data Pipelines Across the Enterprise with Airflow and Control-M
[URL](https://airflowsummit.org/sessions/2026/orchestrating-ai-ready-data-pipelines-across-the-enterprise-with-airflow-and-control-m/)  
Date: 2026-09-01 13:30:00
Summary:
Learn how BMC Software’s Control-M integrates with Airflow to orchestrate end-to-end workflows that connect data pipelines with upstream and downstream enterprise systems such as supply chain, billing, and other mission-critical applications.

## Securing Apache Airflow with Keycloak: A Deep Dive into the Keycloak Auth Manager
[URL](https://airflowsummit.org/sessions/2026/securing-apache-airflow-with-keycloak-a-deep-dive-into-the-keycloak-auth-manager/)  
Date: 2026-09-01 13:30:00
Summary:
As organizations scale their data platforms, managing access to Apache Airflow becomes increasingly complex. In this talk, we introduce the Keycloak Auth Manager — a pluggable authentication and authorization backend for Airflow that delegates identity management to Keycloak, a battle-tested open-source Identity and Access Management solution.
We’ll start with the big picture: what problem does the Keycloak Auth Manager solve, and why Keycloak? We’ll walk through the architecture — how Airflow’s auth manager interface works, how the Keycloak integration hooks into it, and how authentication flows (OIDC/OAuth2) and authorization (role mapping, resource-based permissions) are handled under the hood.


## Stabilizing LinkedIn Continuous Deployment on Airflow
[URL](https://airflowsummit.org/sessions/2026/stabilizing-linkedin-continuous-deployment-on-airflow/)  
Date: 2026-09-01 13:30:00
Summary:
Last year, we showed how LinkedIn’s continuous deployment (LCD) runs on Apache Airflow to orchestrate safe, repeatable releases across thousands of services—powering everyday deployments for 10,000+ engineers.
This year, we’ll dive into the hard‑won patterns that keep those deployments stable at scale: preserving DAG consistency during live updates; routing seamlessly across multiple clusters for graceful failover; enforcing HA guardrails on the control plane; and using dynamic task mapping to deliver faster rollbacks and reduce deployment overhead. You’ll see how we abstract Airflow for a cleaner user experience, what really moved the needle on launching tasks faster, and portable observability practices that cut on‑call toil.


## UX to manage Dags to Assets to Agents - The evolution of the Airflow UI
[URL](https://airflowsummit.org/sessions/2026/ux-to-manage-dags-to-assets-to-agents/)  
Date: 2026-09-01 13:30:00
Summary:
The rise of more complex asset and agentic powered workflows, the Airflow UI needs to evolve beyond just a way to view failed logs and relationships between tasks.
Come see how we are leveraging the latest Airflow features to build new user experiences that can handle growing agentic workflows.
We’ll go through a few workflows to see how they can be solved through a traditional Dag-centric view or a new Asset-centric view. We will also showcase how both are becoming more realtime so you can always see what is happening.


## Managed Workflow for Apache Airflow (MWAA): Latest and Greatest!
[URL](https://airflowsummit.org/sessions/2026/managed-workflow-for-apache-airflow/)  
Date: 2026-09-01 14:00:00
Summary:
MWAA Serverless takes the managed Airflow experience a significant leap forward - eliminating even more operational overhead and allowing teams to focus purely on building and running their workflows. In this session, we’ll explore what makes MWAA Serverless different from MWAA Provisioned, including its auto-scaling compute, pay-per-use pricing model, and zero-infrastructure management, and provide practical guidance on when to choose each option. We’ll also cover the latest enhancements across the MWAA platform. Whether you’re running complex multi-team data pipelines or looking for the simplest path to production Airflow, this session will help you choose the right MWAA deployment model for your workloads. Finally, we’ll dive into Airflow 3.3 and the latest contributions from the MWAA open-source team, including the highly anticipated multi-team support, Deadline Alerts, and upcoming S3 DAG Bundle Versioning. These enhancements demonstrate Amazon’s continued investment in making Apache Airflow more accessible and scalable for the Airflow community.


## Performance Debugging in Airflow: From Symptoms to Solutions
[URL](https://airflowsummit.org/sessions/2026/performance-debugging-in-airflow-from-symptoms-to-solutions/)  
Date: 2026-09-01 14:00:00
Summary:
Airflow running slow? Memory is spiking. Tasks are queuing forever. Now what? Debugging performance issues in a distributed system like Airflow can feel overwhelming—is it the scheduler, the database, the DAG Processor, or your DAG code? This talk shares practical techniques for isolating and fixing performance problems, using real examples from the Airflow codebase.
We’ll cover:
Understanding Airflow’s moving parts – Where bottlenecks typically hide (scheduler loop, DAG parsing, database queries).


## Scaling to 1,000 DAGs: Idelic/Descartes's Blueprint for Airflow Automation and Reliability
[URL](https://airflowsummit.org/sessions/2026/scaling-to-1000-dags/)  
Date: 2026-09-01 14:00:00
Summary:
This session details Idelic/Descartes’s critical journey to a robust, scaled Astronomer Airflow environment. We’ll share technical lessons from overcoming initial orchestration challenges and successfully scaling to over 1,000 active DAGs. The session will showcase our advanced, Jenkins-integrated testing deployment for managing this scale, and the development of a standardized framework that simplifies DAG creation, eliminates code repetition, and enables configuration changes without a full deployment. This is essential for any team managing complex data pipelines, offering a blueprint for standardized Airflow development, maximum data reliability, and future growth at a large scale.


## Apache Airflow, AI Agents and I
[URL](https://airflowsummit.org/sessions/2026/apache-airflow-ai-agents-and-i/)  
Date: 2026-09-01 14:30:00
Summary:
Importance of orchestration grows in AI and agentic area. Now we have both humans and agents authoring, adjusting and troubleshooting Airflow DAGs. What’s more AI agentic capabilities are used inside of Airflow DAGs to tune task logic.
Let’s discuss how Managed Service for Apache Airflow (formerly Cloud Composer) helps in AI agents and users to complete their job.
This session will also provide information about the most recent features of Managed Service for Apache Airflow available to Google Cloud users.


## Self-Service DAGs: Event-Driven Design for GitHub Actions and Airflow at Lyft
[URL](https://airflowsummit.org/sessions/2026/self-service-dags-event-driven-design-for-github-actions-and-airflow-at-lyft/)  
Date: 2026-09-01 14:30:00
Summary:
At Lyft, driver pay configs on GitHub must be validated through Airflow DAGs before merging. However, Scientists and Analysts who change configs are not familiar with Airflow. How do we make such validation self-service while meeting SOX compliance?
This talk presents a design pattern for bidirectional GitHub-Airflow integration: GitHub Actions trigger DAGs, and DAGs push results back as PR status checks via the GitHub Commit Status API. We cover event-driven push-style vs traditional polling style, and why an event-driven push-style works well with Dynamic Task Mapping. This pattern aligns with Airflow 3’s event-driven scheduling vision. We also discuss how SOX requirements shaped this design.


## Taming AI Workloads in Apache Airflow: Dag Patterns to Avoid Infrastructure Instability
[URL](https://airflowsummit.org/sessions/2026/taming-ai-workloads-in-apache-airflow-dag-patterns-to-avoid-infrastructure-instability/)  
Date: 2026-09-01 14:30:00
Summary:
Orchestrating AI workloads introduces a two-front battle with infrastructure instability. First, the Airflow workers themselves (e.g., Kubernetes pod evictions, Celery node scaling) can restart and lose track of active tasks. Second, the external AI cluster running the heavy compute can experience temporary network blips, API timeouts or compute rescheduling. With standard Dag designs, these transient hiccups often cause Airflow to panic, fail the task, and tragically send a kill signal to an expensive, perfectly healthy AI job.


## Dynamic graphs - Airflow for agentic workflows
[URL](https://airflowsummit.org/sessions/2026/dynamic-graphs/)  
Date: 2026-09-01 15:00:00
Summary:
With the growing recognition if the need for agentic orchestration, Airflow is evolving to support a growing set of agentic patterns. Dynamic Task mapping provided a foundation for RAG workflows. Learn how to go beyond those and orchestrate reasoning patterns with dynamic graphs


## Pushing to Prod on a Friday
[URL](https://airflowsummit.org/sessions/2026/pushing-to-prod-on-a-friday/)  
Date: 2026-09-01 15:00:00
Summary:
If the idea of pushing to production on a Friday still makes your stomach drop, you’re in good company because most data professionals know that particular flavor of dread. But that fear says more about systemic fragility than the day of the week. This talk explores how unclear ownership, hidden dependencies, and late validation create production risk in data platforms. I’ll show how data contracts clarify expectations between producers and consumers, how Behavior‑Driven Development (BDD) provides a shared language for system behavior, and how Airflow can enforce guardrails that shift validation earlier and reduce blast radius. This session focuses on the organizational and architectural decisions that shape platform reliability. Because Airflow often becomes the visible surface of upstream uncertainty, its teams feel the impact of broader design and governance choices. Attendees will learn to interpret “Friday fear” as a strategic signal, how contracts and BDD strengthen alignment and predictability, and how Airflow can act as a platform‑level safety system that builds trust and supports confident deployments - Fridays included.


## Scaling Airflow for Capacity Forecasting at Amazon Prime Video
[URL](https://airflowsummit.org/sessions/2026/scaling-airflow-for-capacity-forecasting-at-amazon-prime-video/)  
Date: 2026-09-01 15:00:00
Summary:
Amazon Prime Video uses Airflow to forecast traffic for hundreds of micro-services to deliver the best customer experience for some of the world’s biggest live events across multiple global regions. The forecasting methodology involves complex job dependencies between customer interaction metrics and geographies - translating to ~50 production DAGs with cross-DAG dependencies that process terabytes of customer activity data daily across tens of thousands of compute cores. In this talk, we’ll cover how we manage dependency complexity at scale, coordinate data flows across geographical boundaries, and keep forecasts reliable as the system grows.


## A SQL Query is Just a DAG: Building an SQL Engine on Apache Airflow
[URL](https://airflowsummit.org/sessions/2026/a-sql-query-is-just-a-dag/)  
Date: 2026-09-01 16:00:00
Summary:
Ever wondered what happens between typing SELECT ... GROUP BY and getting results back? Inside every SQL engine lives a scheduler that breaks your query into a DAG of tasks — shuffling, sorting, aggregating, and parallelizing work across partitions. Sound familiar?
In this talk, I’ll demystify SQL engine internals by building one on top of Apache Airflow. We’ll take a SQL query, parse it, optimize it, and transform it into a DAG of Airflow tasks that you can watch execute step by step in the Airflow UI.


## Cloud Composer Workshop - Managing DAGs at Scale
[URL](https://airflowsummit.org/sessions/2026/cloud-composer-workshop-managing-dags-at-scale/)  
Date: 2026-09-01 16:00:00
Summary:
Learn how to effectively set up a CI/CD pipeline that builds, tests, and even corrects (using Gemini) your DAGs before deploying them into your Composer environment.

## Resumable Task Execution for Long Running Tasks such as Spark
[URL](https://airflowsummit.org/sessions/2026/resumable-task-execution-for-long-running-tasks-such-as-spark/)  
Date: 2026-09-01 16:00:00
Summary:
This talk focuses on leveraging the Task State Management (AIP-103) and Enhanced Retry Policy work (AIP-105) being released in Airflow 3.3 to enable enhanced execution of long running tasks including checkpointing, sophisticated (and automated) retry policies, and intra-task observability.
Initially focused on Apache Spark, which is one of the most widely used workload frameworks for data engineers, this is extensible to long running tasks of any type including agentic workflows.


## Shift-Left for Data Pipelines: Improving Airflow Code Quality and Reliability with AI
[URL](https://airflowsummit.org/sessions/2026/shift-left-for-data-pipelines/)  
Date: 2026-09-01 16:00:00
Summary:
Experiment how AI can shift pipeline quality left by enabling earlier detection of issues and improving code reliability during development.

## Stop Being the Dag Bottleneck: How to Scale Airflow Orchestration Beyond Your Engineering Team
[URL](https://airflowsummit.org/sessions/2026/stop-being-the-dag-bottleneck-how-to-scale-airflow-orchestration-beyond-your-engineering-team/)  
Date: 2026-09-01 16:00:00
Summary:
Your data platform team didn’t sign up to be a Dag factory. But when Airflow expertise is concentrated in a small group of engineers, that’s exactly what happens. Analysts wait days for simple workflows, engineers burn cycles rebuilding the same patterns, and frustrated teams start building outside the stack entirely. The real fix isn’t a better onboarding guide or a friendlier UI. It’s rethinking the abstraction layer your team exposes to the rest of the business.


## It Works! Now What? Fast Iteration for AI Capabilities in Airflow
[URL](https://airflowsummit.org/sessions/2026/it-works-now-what-fast-iteration-for-ai-capabilities-in-airflow/)  
Date: 2026-09-01 16:30:00
Summary:
Building an AI capability in Airflow is the easy part. The hard part is what comes next.
You want to swap a model, refactor a prompt, cut token costs, or try a local model instead of paying for cloud. How do you know it still works as expected? Without a fast feedback loop, every change is a gamble.
This talk shows practical patterns for building that feedback loop, with real examples using agent skills, MCPs, and local and cloud models. It covers the challenges too: sandboxing, observability, non-determinism, and keeping checks simple enough that people actually use them.


## The Rise of Abstraction in Dag Authoring: From YAML to Minecraft
[URL](https://airflowsummit.org/sessions/2026/the-rise-of-abstraction-in-dag-authoring-from-yaml-to-minecraft/)  
Date: 2026-09-01 16:30:00
Summary:
In my almost 15 years as a data engineer, I’ve learned one universal truth: everyone needs orchestration. The marketing team needs daily attribution reports. The CRM team needs personalized newsletter triggers. The platform team needs cross-cloud data transfers. The analytics team needs third-party data imports. Data touches every corner of the business, and the orchestration layer is the one layer that connects it all.
This talk explores what becomes possible when we decouple pipeline logic (what happens) from definition (how it’s authored). With the right abstractions, the authoring interface can be anything: Python, declarative YAML, templates, spreadsheets, or even a video game.


## When Airflow Meets Yunikorn: Enhancing Airflow on Kubernetes with Yunikorn for Higher Efficiency
[URL](https://airflowsummit.org/sessions/2026/when-airflow-meets-yunikorn-enhancing-airflow-on-kubernetes-with-yunikorn-for-higher-efficiency/)  
Date: 2026-09-01 16:30:00
Summary:
Apache Airflow’s Kubernetes integration enables flexible workload execution on Kubernetes but lacks advanced resource management features including application queueing, tenant isolation and gang scheduling. These features are increasingly critical for data engineering as well as AI/ML use cases, particularly GPU utilization optimization. For example, gang scheduling ensures all required resources for a job are allocated atomically, preventing partial allocations that waste resources. Apache Yunikorn, a Kubernetes-native scheduler, addresses these gaps by offering a high-performance alternative to Kubernetes default scheduler. In this talk, we’ll demonstrate how to conveniently leverage Yunikorn’s power in Airflow, along with practical use cases and examples.


## Orchestrating and Testing RAG Pipelines with Airflow
[URL](https://airflowsummit.org/sessions/2026/orchestrating-and-testing-rag-pipelines-with-airflow/)  
Date: 2026-09-01 17:00:00
Summary:
RAG pipelines fail silently. Bad retrievals, stale vectors, and unfaithful answers rarely trigger alerts, your row counts pass, your DAG turns green, and your AI product quietly gets worse. This session presents a reference DAG architecture for production-grade RAG ingestion on Airflow 3, with inline quality gates that evaluate retrieval accuracy and answer faithfulness before new vectors are promoted to production. We’ll walk through four failure modes , chunking regressions, embedding model drift, partial re-index states, and retrieval quality decay, and the specific Airflow pattern that catches each, using Ragas for evaluation and Airflow 3’s TaskFlow API, Assets, and DAG versioning for reproducible, event-driven runs.


## Orchestrating GenAI & ML Pipelines with Apache Airflow
[URL](https://airflowsummit.org/sessions/2026/orchestrating-genai-ml-pipelines-with-apache-airflow/)  
Date: 2026-09-01 17:00:00
Summary:
Orchestrating Cross-Account ML & Data Pipelines with Apache Airflow As organizations scale data and ML workloads across multiple AWS accounts and Regions, orchestration becomes the hardest engineering problem — not the models themselves. This session shows how Apache Airflow serves as a centralized orchestration hub for distributed data-processing and machine-learning pipelines that span account and regional boundaries.
We walk through a production-ready architecture where a single Airflow environment coordinates:
Cross-account DAG patterns — using Airflow connections, IAM role assumption, and custom hooks to trigger AWS Glue, SageMaker, and Lambda in remote accounts


## Remote Control Isolation: airflowctl becomes the new default
[URL](https://airflowsummit.org/sessions/2026/remote-control-isolation-airflowctl-becomes-the-new-default/)  
Date: 2026-09-01 17:00:00
Summary:
Meet airflowctl, the new default for API-driven remote operations. You will see how separating control from execution enhances security, enables isolation, and simplifies automation across different environments. I will discuss the development of airflowctl, demonstrate practical examples of secure remote execution, and provide a guide for transitioning from legacy workflows. You will learn how to easily migrate towards airflowctl and leverage the flexibility of an API-driven approach.


## One Codebase, Many Distributions: Airflow’s Modular Approach
[URL](https://airflowsummit.org/sessions/2026/one-codebase-many-distributions-airflows-modular-approach/)  
Date: 2026-09-01 17:30:00
Summary:
Airflow’s evolution toward a client-server architecture faced a fundamental challenge: splitting a monolithic codebase into independent distributions (airflow-core, task-sdk, providers) without triggering dependency hell. Traditional PyPi packaging and code duplication both fail at Airflow’s scale.
Airflow 3.2 solves this through modular isolation and shared libraries using in-repository symlinks. This approach ensures each distribution ships with the exact version of shared code it requires, eliminating runtime version conflicts and allowing for independent dependency management. We have already migrated 10+ critical components—including the config parser, observability, and secrets masking—into this shared model.


## A Decade of Spark + Airflow: How Two Communities Built the De Facto Standard for Data Engineering
[URL](https://airflowsummit.org/sessions/2026/a-decade-of-spark-airflow-how-two-communities-built-the-de-facto-standard-for-data-engineering/)  
Date: 2026-09-02 10:30:00
Summary:
No two open source projects have shaped modern data engineering more than Apache Spark and Apache Airflow. But their partnership wasn’t designed- it was earned. From the early days of BashOperator wrapping spark-submit, through the SparkSubmitOperator, Livy, Kubernetes-native execution, and now Airflow 3’s asset-aware scheduling paired with Spark’s Declarative Pipelines, the integration story is a masterclass in how independent communities converge on shared problems without shared governance. This talk traces the full arc: how Spark’s compute model and Airflow’s orchestration model co-evolved, where they fought, where they complemented each other, and what the next chapter looks like as both projects ship their most ambitious releases simultaneously. Along the way, we’ll examine the contribution patterns, the cross-pollination of committers, and why this particular pairing outlasted every managed alternative that tried to replace it. This is not a vendor talk. This is a community talk about what happens when two ecosystems trust each other enough to stay independent.


## Airflow 3 as a Dungeon Master: Orchestrating a Text Adventure
[URL](https://airflowsummit.org/sessions/2026/airflow-3-as-a-dungeon-master/)  
Date: 2026-09-02 10:30:00
Summary:
A live text adventure where Airflow is the game engine. Rooms are tasks. Choices are branches. Inventory lives in XComs. Monsters have Deadline Alerts. The audience votes at every fork, and the DAG decides what happens next. It’s silly, it’s live, and every concept maps to a real production pattern.


## From Broken DAGs to Self-Healing Pipelines: AI-Driven Orchestration with Airflow
[URL](https://airflowsummit.org/sessions/2026/from-broken-dags-to-self-healing-pipelines/)  
Date: 2026-09-02 10:30:00
Summary:
Data pipelines often fail due to upstream issues, infrastructure instability, or unexpected spikes in data volume. While Apache Airflow provides retries and alerts, many failures still require manual intervention. As data platforms scale, this reactive approach becomes difficult to manage.
In this session, we will explore how teams can design self-healing data pipelines using Airflow combined with AI-driven insights. We will look at common pipeline failure patterns and how anomaly detection on metrics such as task runtimes, retries, and data volume changes can help identify issues early. By combining intelligent monitoring with automated recovery strategies, engineers can reduce downtime and operational overhead while building more resilient and reliable data workflows.


## Get Certified: DAG Authoring for Apache Airflow 3
[URL](https://airflowsummit.org/sessions/2026/get-certified-dag-authoring-for-apache-airflow-3/)  
Date: 2026-09-02 10:30:00
Summary:
Earn your DAG Authoring certification in person with a pre-exam preparation and Q&A.

## Toward a Polyglot Airflow
[URL](https://airflowsummit.org/sessions/2026/toward-a-polyglot-airflow/)  
Date: 2026-09-02 10:30:00
Summary:
Building on Airflow 3’s new worker structure and foundation laied by Go SDK, we take a look at how Airflow can support a fully cross-language Dag-authoring experience.
We will discuss how a new language SDK is built, how a task talks to Airflow, and how multiple languages may be mixed inside a Dag. To support additional languages without logic duplication, a new middle layer is required between Airflow and the task. Additional topics, such as security, distributed workload, and user interface considerations, will also be touched on.


## Your first Apache Airflow Contribution
[URL](https://airflowsummit.org/sessions/2026/your-first-apache-airflow-contribution/)  
Date: 2026-09-02 10:30:00
Summary:
Ready to contribute to Apache Airflow? In this hands-on workshop, we’ll help you jump straight into the project with real, beginner-friendly issues matched to your skills and interests.

## Agor: A Collaborative Orchestration Layer for AI Agents
[URL](https://airflowsummit.org/sessions/2026/agor-a-collaborative-orchestration-layer/)  
Date: 2026-09-02 11:00:00
Summary:
Agor is an open-source platform for orchestrating AI agents: built for teams, not just individuals. It provides a shared, real-time workspace where humans and agents collaborate on a spatial canvas. Multiple agents run in parallel across isolated git worktrees, with full visibility into sessions, conversations, and outputs. Teams can inspect, intervene, and steer work as it happens.
At the core are persistent assistants: long-lived agents with memory and tools that coordinate tasks, spawn sub-agents, and continuously advance workflows.


## Talk to your Dags: Airflow AI Assistant (AIP-101 proposal)
[URL](https://airflowsummit.org/sessions/2026/talk-to-your-dags-airflow-ai-assistant-aip-101-proposal/)  
Date: 2026-09-02 11:00:00
Summary:
Airflow runs the pipelines that matter. When a task breaks, the workflow is fragmented: you copy an error, ask your favorite LLM in another tab, and still end up back in the Grid, scrolling logs. AIP-101 proposes a better way: an opt-in AI assistant, natively integrated into Apache Airflow. Ask about your Dags, runs, and logs, and get grounded answers based on what you can already see. Built with a safety-first mindset, it respects your existing access, keeps sensitive details out of responses, and makes its help transparent. In this initial phase, the assistant explains, not executes. This talk highlights the user experience, the key design decisions, suggested high-level architecture, and what comes next for AI in Airflow.


## Triggers at Datadog: What Are Trigger Queues and Why You Should Use Them
[URL](https://airflowsummit.org/sessions/2026/triggers-at-datadog-what-are-trigger-queues-and-why-you-should-use-them/)  
Date: 2026-09-02 11:00:00
Summary:
Datadog is a world-class data platform ingesting more than a 100 trillion events a day, providing real-time insights. Since our internal adoption of Airflow following the release of 3.0.0, the number of teams relying on our internal Airflow platform have grown organically and quickly.
This internal Airflow adoption came with a number of platform challenges, requiring novel solutions which could support multi-tenancy, scalability, and bespoke runtime environments. In this talk, we will cover how we’ve expanded the functionality of Airflow triggers – via trigger queue assignment – to support multi-tenancy deployments, while contributing those solutions upstream with the broader Airflow community. We’ll cover the conceptual design and motivations for Trigger queues, and how the trigger queue pattern can benefit both multi-tenant and single-occupant Airflow systems alike.


## Airflow Already Knows the Root Cause
[URL](https://airflowsummit.org/sessions/2026/airflow-already-knows-root-cause/)  
Date: 2026-09-02 11:30:00
Summary:
Dashboards show symptoms. Lineage shows relationships. But neither explains why something happened.
In the AI era, Airflow is no longer just an orchestrator - it’s a source of operational intelligence. OpenLineage, the open standard for data lineage, emits rich, structured context across Airflow, Spark, and other tools - including failure logs and detailed execution metadata.
With recent improvements - out-of-the-box run ID correlation between Airflow entities, powerful hook-level lineage for SQL operators (capturing queries and query IDs from within Python operators), human-in-the-loop metadata (who approved what and when), and more - the context layer is more powerful than ever.


## Beyond Multi-Cluster Airflow: Operating GPU Workloads at Scale
[URL](https://airflowsummit.org/sessions/2026/beyond-multi-cluster-airflow-operating-gpu-workloads-at-scale/)  
Date: 2026-09-02 11:30:00
Summary:
At last year’s Airflow Summit, we shared how we built a multi-cluster orchestration layer on top of Apache Airflow to run ML workloads across multiple Kubernetes GPU clusters.
Once hundreds of ML engineers started running GPU pipelines in production, we discovered that orchestration alone is not enough. Operating multi-cluster GPU infrastructure introduces new challenges: controlling GPU allocation across teams, observing pipelines across clusters, and helping users run workloads efficiently without wasting expensive GPU resources.


## From JAR to DAG: Running Java Tasks Natively with the Airflow Java-SDK
[URL](https://airflowsummit.org/sessions/2026/from-jar-to-dag/)  
Date: 2026-09-02 11:30:00
Summary:
Airflow is Python-first — but production business logic is often Java-first. The Airflow Java SDK bridges that gap by letting you mix Java and Python tasks within the same DAG, without shell wrappers or separate services.
In this talk, we’ll walk through the full lifecycle of a Java task: how the Java SDK is set up, how tasks are defined and packaged as a JAR, how Airflow picks them up and runs them on any executor, and how results flow back into your DAG. We’ll also cover how core Airflow primitives — Variables, Connections, XCom, and logging — work natively in the Java SDK, enabling true cross-language, bidirectional communication within a single pipeline. You’ll see it all running end-to-end in a live demo alongside Python tasks.


## Agentic Incident DAG: Structured, Replayable Data Incident Intelligence
[URL](https://airflowsummit.org/sessions/2026/agentic-incident-dag/)  
Date: 2026-09-02 12:00:00
Summary:
Data incidents are often investigated through fragmented Slack threads and manual SQL queries, leaving data owners dependent on engineers. Qbiz introduces a more efficient alternative: the Agentic Incident DAG. This approach uses AI agents to lead investigations while Airflow orchestrates a systematic diagnostic workflow.
When a failure occurs, the system triggers a diagnostic DAG and creates a Data Incident Ticket. An Investigation Thread captures the analysis in real time as specialized agents evaluate potential causes and provide clear summaries for data owners.


## From UC4 Automic to Airflow at eBay: Bridging the Enterprise Scheduler Feature Gap at Scale
[URL](https://airflowsummit.org/sessions/2026/from-uc4-automic/)  
Date: 2026-09-02 12:00:00
Summary:
Migrating from UC4 Automic to Apache Airflow is far from a lift-and-shift exercise. UC4 offers advanced scheduling primitives that data teams rely on daily — and Airflow doesn’t replicate them out of the box.
At eBay, we migrated thousands of business-critical UC4 workflows onto our Airflow 2.10 platform. Rather than forcing teams to change how they operate, we built the missing capabilities natively into Airflow:
Breakpoints — pause a pipeline at a specific task for inspection without failing the run Skip logic — dynamically bypass tasks or task groups at runtime Calendar-aware scheduling — replicate UC4’s calendar model as custom Airflow timetables Pipeline pause/resume — operator-triggered suspension of in-flight DAG runs with state consistency We’ll share the engineering trade-offs, architectural constraints we hit, and patterns reusable beyond eBay’s stack.


## Zero‑Code Airflow at Scale: Building Enterprise Pipelines from CSV to Production DAGs
[URL](https://airflowsummit.org/sessions/2026/zero-code-airflow/)  
Date: 2026-09-02 12:00:00
Summary:
As Airflow adoption expands across large enterprises, a core challenge emerges: How to enable multiple teams to design and operate data pipelines without relying heavily on specialized engineering expertise. In this session, we will present a zero‑code, metadata‑driven Airflow framework built and deployed within a large financial services organization to accelerate pipeline development and onboarding at scale.
This framework allows users to define workflows using simple CSV or Excel inputs, which are automatically converted into YAML configurations and deployed as fully production‑ready Airflow DAGs using standardized templates on Astronomer. By leveraging a remote execution model and reusable DAG patterns, the solution supports orchestration across heterogeneous systems—including data warehouses, ingestion pipelines, and data quality frameworks—while maintaining enterprise‑grade governance, consistency, and observability.


## Lightning talks + Lunch
[URL](https://airflowsummit.org/sessions/2026/lightning-talks/)  
Date: 2026-09-02 12:30:00
Summary:
Fellow Airflow Summit participants get 5 minutes to share things of interest to the audience, whether it is something about a project you are working on, something you learned at Airflow Summit, advice for fellow data engineers, etcetera. We will be providing instructions on how to submit your talk closer to the event.
Lunch will be served during this activity.


## Community discussions: Airflow
[URL](https://airflowsummit.org/sessions/2026/community-airflow/)  
Date: 2026-09-02 13:30:00
Summary:
This time will be used for community discussions and activities related to Apache Airflow.


## Community discussions: Spark
[URL](https://airflowsummit.org/sessions/2026/community-spark/)  
Date: 2026-09-02 13:30:00
Summary:
This time will be used for community discussions and activities related to Apache Spark.


## Community discussions: TBD
[URL](https://airflowsummit.org/sessions/2026/community-tbd1/)  
Date: 2026-09-02 13:30:00
Summary:
This time will be used for community discussions and activities related to Apache Spark.


## Orchestrating 100 ML Models using Airflow
[URL](https://airflowsummit.org/sessions/2026/orchestrating-100-ml-models-using-airflow/)  
Date: 2026-09-02 9:00:00
Summary:
Productionizing ML workflows is complicated; scaling them is harder. At Ramp, we grew from zero to nearly 100 production ML models powering systems like credit risk assessment and sales lead valuation.
This talk covers how Airflow became the backbone of our ML platform, orchestrating ETL jobs, data quality checks, and model runs. We’ll discuss how we evolved it to meet the increasing complexity of our ML systems.
Every ML system consists of feature creation and large-batch inference. We started with a few DBT models and one cloud-hosted notebook, which evolved into thousands of upstream tables and hundreds of AWS batch inference jobs.


## Autonomous, Not Unsupervised: Agent-Authored Pipelines on Airflow at Together AI
[URL](https://airflowsummit.org/sessions/2026/autonomous-not-unsupervised/)  
Date: 2026-09-02 9:30:00
Summary:
At Together AI, AI agents have become the primary authors of our production data pipelines — and Airflow is what makes that safe to do. Agents do the building. Airflow gives us the surface to set the rules, see what’s happening, and step in when we need to. The interesting part is what each side has to look like for that to actually work in production. This talk is a field report on that relationship. We’ll walk how we got from a world where humans wrote SQL by hand and dashboards refreshed nightly to one where agents make hundreds of queries per session, catalog thousands of tables across engines, and ship pipelines in hours instead of weeks. The platform now spans twelve dbt projects across billing, inference, and analytics — all of it agent-authored, all of it running through Airflow.


# SECTION: Speakers



## Abdul Majid Mohammed
[URL](https://airflowsummit.org/speakers/abdul-majid-mohammed/)

## Abhishek Bhakat
[URL](https://airflowsummit.org/speakers/abhishek-bhakat/)

## Ace Haidrey
[URL](https://airflowsummit.org/speakers/ace-haidrey/)

## Adam Boscarino
[URL](https://airflowsummit.org/speakers/adam-boscarino/)

## Ahmad Farhan
[URL](https://airflowsummit.org/speakers/ahmad-farhan/)

## Ahmed Chakir Alaoui
[URL](https://airflowsummit.org/speakers/ahmed-chakir/)

## Ahuitz Rojas
[URL](https://airflowsummit.org/speakers/ahuitz-rojas/)

## Aishwarya Sankaravadivel
[URL](https://airflowsummit.org/speakers/aishwarya-sankaravadivel/)

## Aizhamal Nurmamat kyzy
[URL](https://airflowsummit.org/speakers/aizhamal-nurmamat/)

## Ajay Kumar Bhupatiraju
[URL](https://airflowsummit.org/speakers/ajay-kumar-bhupatiraju/)

## Akanksha Khushboo
[URL](https://airflowsummit.org/speakers/akanksha-khushboo/)

## Akshay Battaje
[URL](https://airflowsummit.org/speakers/akshay-battaje/)

## Alaeddine Maaoui
[URL](https://airflowsummit.org/speakers/alaeddine-maaoui/)

## Alan Chin
[URL](https://airflowsummit.org/speakers/alan-chin/)

## Aldo Orozco Gomez
[URL](https://airflowsummit.org/speakers/aldo-orozco-gomez/)

## Aleksandr Shirokov
[URL](https://airflowsummit.org/speakers/aleksandr-shirokov/)

## Alessandro Pregnolato
[URL](https://airflowsummit.org/speakers/alessandro-pregnolato/)

## Alex Guglielmone
[URL](https://airflowsummit.org/speakers/alex-guglielmone/)

## Alexander Eliseev
[URL](https://airflowsummit.org/speakers/alexander-eliseev/)

## Alexander Shmidt
[URL](https://airflowsummit.org/speakers/alexander-shmidt/)

## Alexandra Abbas
[URL](https://airflowsummit.org/speakers/alexandra-abbas/)

## Alida Laney
[URL](https://airflowsummit.org/speakers/alida-laney/)

## Alina Aven
[URL](https://airflowsummit.org/speakers/alina-aven/)

## Alvin Zhang
[URL](https://airflowsummit.org/speakers/alvin-zhang/)

## Amin Abou-Gallala
[URL](https://airflowsummit.org/speakers/amin-abou-gallala/)

## Amit Chauhan
[URL](https://airflowsummit.org/speakers/amit-chauhan/)

## Amit Kumar
[URL](https://airflowsummit.org/speakers/amit-kumar/)

## Amogh Rajesh Desai
[URL](https://airflowsummit.org/speakers/amogh-desai/)

## Amr Noureldin
[URL](https://airflowsummit.org/speakers/amr-noureldin/)

## Amy Pitcher
[URL](https://airflowsummit.org/speakers/amy-pitcher/)

## Anandhi Murali
[URL](https://airflowsummit.org/speakers/anandhi-murali/)

## Anant Agarwal
[URL](https://airflowsummit.org/speakers/anant-agarwal/)

## Andrea Bombino
[URL](https://airflowsummit.org/speakers/andrea-bombino/)

## Andreas Neumann
[URL](https://airflowsummit.org/speakers/andreas-neumann/)

## Andres Astorga Espriella
[URL](https://airflowsummit.org/speakers/andres-astorga-espriella/)

## Andrew Godwin
[URL](https://airflowsummit.org/speakers/andrew-godwin/)

## Aneel Murari
[URL](https://airflowsummit.org/speakers/aneel-murari/)

## Angel Daz
[URL](https://airflowsummit.org/speakers/angel-daz/)

## Aniruddha Sengupta
[URL](https://airflowsummit.org/speakers/aniruddha-sengupta/)

## Anish Giri
[URL](https://airflowsummit.org/speakers/anish-giri/)

## Anita Fronczak
[URL](https://airflowsummit.org/speakers/anita-fronczak/)

## Ankit Chaurasia
[URL](https://airflowsummit.org/speakers/ankit-chaurasia/)

## Annie Friedman
[URL](https://airflowsummit.org/speakers/annie-friedman/)

## Annie Kaufman
[URL](https://airflowsummit.org/speakers/annie-kaufman/)

## Anshal Raj
[URL](https://airflowsummit.org/speakers/anshal-raj/)

## Anthony Figueroa
[URL](https://airflowsummit.org/speakers/anthony-figueroa/)

## Anthony Kalsatos
[URL](https://airflowsummit.org/speakers/anthony-kalsatos/)

## Anton Ivanov
[URL](https://airflowsummit.org/speakers/anton-ivanov/)

## Anum Sheraz
[URL](https://airflowsummit.org/speakers/anum-sheraz/)

## Anurag Srivastava
[URL](https://airflowsummit.org/speakers/anurag-srivastava/)

## Aritra Ghosh
[URL](https://airflowsummit.org/speakers/aritra-ghosh/)

## Arthur Chen
[URL](https://airflowsummit.org/speakers/arthur-chen/)

## Arun Joy Vattoly
[URL](https://airflowsummit.org/speakers/arun-joy-vattoly/)

## Arun Kumar
[URL](https://airflowsummit.org/speakers/arun-kumar/)

## Arunav Gupta
[URL](https://airflowsummit.org/speakers/arunav-gupta/)

## Ash Berlin-Taylor
[URL](https://airflowsummit.org/speakers/ash-berlin-taylor/)

## Ashim Shrestha
[URL](https://airflowsummit.org/speakers/ashim-shrestha/)

## Ashir Alam
[URL](https://airflowsummit.org/speakers/ashir-alam/)

## Ashley Gough
[URL](https://airflowsummit.org/speakers/ashley-gough/)

## Ashok Prakash
[URL](https://airflowsummit.org/speakers/ashok-prakash/)

## Augusto Hidalgo
[URL](https://airflowsummit.org/speakers/augusto-hidalgo/)

## Austin Bennett
[URL](https://airflowsummit.org/speakers/austin-bennett/)

## Ayoade Adegbite
[URL](https://airflowsummit.org/speakers/ayoade-adegbite/)

## Bao Nguyen
[URL](https://airflowsummit.org/speakers/bao-nguyen/)

## Barr Moses
[URL](https://airflowsummit.org/speakers/barr-moses/)

## Bartosz Jankiewicz
[URL](https://airflowsummit.org/speakers/bartosz-jankiewicz/)

## Bartłomiej Hirsz
[URL](https://airflowsummit.org/speakers/bart%C5%82omiej-hirsz/)

## Bas Harenslak
[URL](https://airflowsummit.org/speakers/bas-harenslak/)

## Basil Faruqui
[URL](https://airflowsummit.org/speakers/basil-faruqui/)

## Belle Romea
[URL](https://airflowsummit.org/speakers/belle-romea/)

## Ben Chen
[URL](https://airflowsummit.org/speakers/ben-chen/)

## Ben Rogojan
[URL](https://airflowsummit.org/speakers/ben-rogojan/)

## Bhavani Ravi
[URL](https://airflowsummit.org/speakers/bhavani-ravi/)

## Bhavesh Jaisinghani
[URL](https://airflowsummit.org/speakers/bhavesh-jaisinghani/)

## Blagoy Kaloferov
[URL](https://airflowsummit.org/speakers/blagoy-kaloferov/)

## Blaine Elliot
[URL](https://airflowsummit.org/speakers/blaine-elliot/)

## Boaz Farkash
[URL](https://airflowsummit.org/speakers/boaz-farkash/)

## Bob Reno
[URL](https://airflowsummit.org/speakers/bob-reno/)

## Bolke de Bruin
[URL](https://airflowsummit.org/speakers/bolke-de-bruin/)

## Bonnie Why
[URL](https://airflowsummit.org/speakers/bonnie-why/)

## Bowrna Prabhakaran
[URL](https://airflowsummit.org/speakers/bowrna-prabhakaran/)

## Brad Kirn
[URL](https://airflowsummit.org/speakers/brad-kirn/)

## Branden West
[URL](https://airflowsummit.org/speakers/branden-west/)

## Brandon Abear
[URL](https://airflowsummit.org/speakers/brandon-abear/)

## Brent Bovenzi
[URL](https://airflowsummit.org/speakers/brent-bovenzi/)

## Briana Okyere
[URL](https://airflowsummit.org/speakers/briana-okyere/)

## Brooke Jamieson
[URL](https://airflowsummit.org/speakers/brooke-jamieson/)

## Bryan Johns
[URL](https://airflowsummit.org/speakers/bryan-johns/)

## Buğra Öztürk
[URL](https://airflowsummit.org/speakers/bugra-ozturk/)

## Burak Özdemir
[URL](https://airflowsummit.org/speakers/burak-ozdemir/)

## Caitlin Petro
[URL](https://airflowsummit.org/speakers/caitlin-petro/)

## Cedrik Neumann
[URL](https://airflowsummit.org/speakers/cedrik-neumann/)

## Chandra Kiran Yelagam
[URL](https://airflowsummit.org/speakers/chandra-kiran-yelagam/)

## Charles Adetiloye
[URL](https://airflowsummit.org/speakers/charles-adetiloye/)

## Chhayank Jain
[URL](https://airflowsummit.org/speakers/chhayank-jain/)

## Chinni Krishna Abburi
[URL](https://airflowsummit.org/speakers/chinni-krishna-abburi/)

## Chirag Tailor
[URL](https://airflowsummit.org/speakers/chirag-tailor/)

## Chirag Todarka
[URL](https://airflowsummit.org/speakers/chirag-todarka/)

## Christian Foernges
[URL](https://airflowsummit.org/speakers/christian-foernges/)

## Christian Schilling
[URL](https://airflowsummit.org/speakers/christian-schilling/)

## Christian Yarros
[URL](https://airflowsummit.org/speakers/christian-yarros/)

## Christine Shen
[URL](https://airflowsummit.org/speakers/christine-shen/)

## Christos Bisias
[URL](https://airflowsummit.org/speakers/christos-bisias/)

## C.J. Jameson
[URL](https://airflowsummit.org/speakers/cj-jameson/)

## Clayton Coleman
[URL](https://airflowsummit.org/speakers/clayton-coleman/)

## Collin Mcnulty
[URL](https://airflowsummit.org/speakers/collin-mcnulty/)

## Cong Zhu
[URL](https://airflowsummit.org/speakers/cong-zhu/)

## Conor Camp
[URL](https://airflowsummit.org/speakers/conor-camp/)

## Constance Martineau
[URL](https://airflowsummit.org/speakers/constance-martineau/)

## Corrine Tan
[URL](https://airflowsummit.org/speakers/corrine-tan/)

## Cyrus Dukart
[URL](https://airflowsummit.org/speakers/cyrus-dukart/)

## DaeHoon Song
[URL](https://airflowsummit.org/speakers/daehoon-song/)

## Dan Davydov
[URL](https://airflowsummit.org/speakers/dan-davydov/)

## Daniel Imberman
[URL](https://airflowsummit.org/speakers/daniel-imberman/)

## Daniel Standish
[URL](https://airflowsummit.org/speakers/daniel-standish/)

## Daniel Wolf
[URL](https://airflowsummit.org/speakers/daniel-wolf/)

## Daniil Dubin
[URL](https://airflowsummit.org/speakers/daniil-dubin/)

## Danny De Leo
[URL](https://airflowsummit.org/speakers/danny-de-leo/)

## Danny Tarlow
[URL](https://airflowsummit.org/speakers/danny-tarlow/)

## Dave Milmont
[URL](https://airflowsummit.org/speakers/dave-milmont/)

## David Sacerdote
[URL](https://airflowsummit.org/speakers/david-sacerdote/)

## David Sevier
[URL](https://airflowsummit.org/speakers/david-sevier/)

## David Xue
[URL](https://airflowsummit.org/speakers/david-xue/)

## Debasish Das
[URL](https://airflowsummit.org/speakers/debasish-das/)

## Deepan Ignaatious
[URL](https://airflowsummit.org/speakers/deepan-ignaatious/)

## Dennis Ferruzzi
[URL](https://airflowsummit.org/speakers/dennis-ferruzzi/)

## Deva Madhavan
[URL](https://airflowsummit.org/speakers/deva-madhavan/)

## Dheeraj Turaga
[URL](https://airflowsummit.org/speakers/dheeraj-turaga/)

## Diana Vazquez Romo
[URL](https://airflowsummit.org/speakers/diana-vazquez-romo/)

## Diederik van Liere
[URL](https://airflowsummit.org/speakers/diederik-van-liere/)

## Diego Zimmerman
[URL](https://airflowsummit.org/speakers/diego-zimmerman/)

## Dinghang Yu
[URL](https://airflowsummit.org/speakers/dinghang-yu/)

## Dipankar Ghosal
[URL](https://airflowsummit.org/speakers/dipankar-ghosal/)

## Dmitry Suvorov
[URL](https://airflowsummit.org/speakers/dmitry-suvorov/)

## Dolev Farhi
[URL](https://airflowsummit.org/speakers/dolev-farhi/)

## Doron Cohen
[URL](https://airflowsummit.org/speakers/doron-cohen/)

## Doug Rogan
[URL](https://airflowsummit.org/speakers/doug-rogan/)

## Dustin Ingram
[URL](https://airflowsummit.org/speakers/dustin-ingram/)

## Ed Sparkes
[URL](https://airflowsummit.org/speakers/ed-sparkes/)

## Eden Gluska
[URL](https://airflowsummit.org/speakers/eden-gluska/)

## Elad Kalif
[URL](https://airflowsummit.org/speakers/elad-kalif/)

## Elad Yaniv
[URL](https://airflowsummit.org/speakers/elad-yaniv/)

## Eli Scheele
[URL](https://airflowsummit.org/speakers/eli-scheele/)

## Eloi Codina Torras
[URL](https://airflowsummit.org/speakers/eloi-codina-torras/)

## Elona Zharri
[URL](https://airflowsummit.org/speakers/elona-zharri/)

## Emil Todorov
[URL](https://airflowsummit.org/speakers/emil-todorov/)

## Ephraim Anierobi
[URL](https://airflowsummit.org/speakers/ephraim-anierobi/)

## Eric Jones
[URL](https://airflowsummit.org/speakers/eric-jones/)

## Ethan Shalev
[URL](https://airflowsummit.org/speakers/ethan-shalev/)

## Ethan Stone
[URL](https://airflowsummit.org/speakers/ethan-stone/)

## Ethan (Tianyang) Lin
[URL](https://airflowsummit.org/speakers/ethan-tianyang-lin/)

## Euccas Chen
[URL](https://airflowsummit.org/speakers/euccas-chen/)

## Evan Tahler
[URL](https://airflowsummit.org/speakers/evan-tahler/)

## Evgeny Nuger
[URL](https://airflowsummit.org/speakers/evgeny-nuger/)

## Evgeny Shulman
[URL](https://airflowsummit.org/speakers/evgeny-shulman/)

## Filip Knapik
[URL](https://airflowsummit.org/speakers/filip-knapik/)

## Filip Kunčar
[URL](https://airflowsummit.org/speakers/filip-kuncar/)

## Francisco Alberini
[URL](https://airflowsummit.org/speakers/francisco-alberini/)

## Freddy Demiane
[URL](https://airflowsummit.org/speakers/freddy-demiane/)

## Fritz Davenport
[URL](https://airflowsummit.org/speakers/fritz-davenport/)

## Gabby Clavell
[URL](https://airflowsummit.org/speakers/gabby-clavell/)

## Gangfeng Huang
[URL](https://airflowsummit.org/speakers/gangfeng-huang/)

## Gaëtan Podevijn
[URL](https://airflowsummit.org/speakers/gaetan-podevijn/)

## George Yates
[URL](https://airflowsummit.org/speakers/george-yates/)

## Gerard Casas Saez
[URL](https://airflowsummit.org/speakers/gerard-casas-saez/)

## Gil Reich
[URL](https://airflowsummit.org/speakers/gil-reich/)

## Gonçalo Costa
[URL](https://airflowsummit.org/speakers/goncalo-costa/)

## Greg Neiheisel
[URL](https://airflowsummit.org/speakers/greg-neiheisel/)

## Griselda Cuevas
[URL](https://airflowsummit.org/speakers/griselda-cuevas/)

## Gunnar Lykins
[URL](https://airflowsummit.org/speakers/gunnar-lykins/)

## Gurmeet Saran
[URL](https://airflowsummit.org/speakers/gurmeet-saran/)

## Hamed Saljooghinejad
[URL](https://airflowsummit.org/speakers/hamed-saljooghinejad/)

## Han Gan
[URL](https://airflowsummit.org/speakers/han-gan/)

## Hannah Lundrigan
[URL](https://airflowsummit.org/speakers/hannah-lundrigan/)

## Haofei Feng
[URL](https://airflowsummit.org/speakers/haofei-feng/)

## Harel Shein
[URL](https://airflowsummit.org/speakers/harel-shein/)

## Hari Nair
[URL](https://airflowsummit.org/speakers/hari-nair/)

## Hendrik Kleine
[URL](https://airflowsummit.org/speakers/hendrik-kleine/)

## Hoa Nguyen
[URL](https://airflowsummit.org/speakers/hoa-nguyen/)

## Howard Yoo
[URL](https://airflowsummit.org/speakers/howard-yoo/)

## Howie Wang
[URL](https://airflowsummit.org/speakers/howie-wang/)

## Huanjie Guo
[URL](https://airflowsummit.org/speakers/huanjie-guo/)

## Huiliang Zhang
[URL](https://airflowsummit.org/speakers/huiliang-zhang/)

## Hussein Awala
[URL](https://airflowsummit.org/speakers/hussein-awala/)

## Ian Moritz
[URL](https://airflowsummit.org/speakers/ian-moritz/)

## Iddo Avneri
[URL](https://airflowsummit.org/speakers/iddo-avneri/)

## Igor Kholopov
[URL](https://airflowsummit.org/speakers/igor-kholopov/)

## Ipsa Trivedi
[URL](https://airflowsummit.org/speakers/ipsa-trivedi/)

## Israel Herraiz
[URL](https://airflowsummit.org/speakers/israel-herraiz/)

## Itai Yaffe
[URL](https://airflowsummit.org/speakers/itai-yaffe/)

## Ivan Nikolov
[URL](https://airflowsummit.org/speakers/ivan-nikolov/)

## Ivan Sayapin
[URL](https://airflowsummit.org/speakers/ivan-sayapin/)

## Jack Cusick
[URL](https://airflowsummit.org/speakers/jack-cusick/)

## Jack Lockyer-Stevens
[URL](https://airflowsummit.org/speakers/jack-lockyer-stevens/)

## Jacob Ferriero
[URL](https://airflowsummit.org/speakers/jacob-ferriero/)

## Jakub Orłowski
[URL](https://airflowsummit.org/speakers/jakub-orlowski/)

## Jan Pawłowski
[URL](https://airflowsummit.org/speakers/jan-pawlowski/)

## Jarek Potiuk
[URL](https://airflowsummit.org/speakers/jarek-potiuk/)

## Jason Bridgemohansingh
[URL](https://airflowsummit.org/speakers/jason-bridgemohansingh/)

## Jason Ma
[URL](https://airflowsummit.org/speakers/jason-ma/)

## Jay Sen
[URL](https://airflowsummit.org/speakers/jay-sen/)

## Jay Thomas
[URL](https://airflowsummit.org/speakers/jay-thomas/)

## Jed Cunningham
[URL](https://airflowsummit.org/speakers/jed-cunningham/)

## Jeetendra Vaidya
[URL](https://airflowsummit.org/speakers/jeetendra-vaidya/)

## Jeff Zhang
[URL](https://airflowsummit.org/speakers/jeff-zhang/)

## Jelle Munk
[URL](https://airflowsummit.org/speakers/jelle-munk/)

## Jen Dunham
[URL](https://airflowsummit.org/speakers/jen-dunham/)

## Jennifer Chisik
[URL](https://airflowsummit.org/speakers/jennifer-chisik/)

## Jennifer Melot
[URL](https://airflowsummit.org/speakers/jennifer-melot/)

## Jenny Gao
[URL](https://airflowsummit.org/speakers/jenny-gao/)

## Jens Scheffler
[URL](https://airflowsummit.org/speakers/jens-scheffler/)

## Jeremy Beard
[URL](https://airflowsummit.org/speakers/jeremy-beard/)

## Jet Mariscal
[URL](https://airflowsummit.org/speakers/jet-mariscal/)

## Jianlong Zhong
[URL](https://airflowsummit.org/speakers/jianlong-zhong/)

## Jiayu Yi
[URL](https://airflowsummit.org/speakers/jiayu-yi/)

## Jilan Kothakota
[URL](https://airflowsummit.org/speakers/jilan-kothakota/)

## Jim Howard
[URL](https://airflowsummit.org/speakers/jim-howard/)

## Jitendra Jaladi
[URL](https://airflowsummit.org/speakers/jitendra-jaladi/)

## Jitendra Shah
[URL](https://airflowsummit.org/speakers/jitendra-shah/)

## John Jackson
[URL](https://airflowsummit.org/speakers/john-jackson/)

## John Joyce
[URL](https://airflowsummit.org/speakers/john-joyce/)

## John Robert
[URL](https://airflowsummit.org/speakers/john-robert/)

## Jon Fink
[URL](https://airflowsummit.org/speakers/jon-fink/)

## Jon Hiett
[URL](https://airflowsummit.org/speakers/jon-hiett/)

## Jonathan Lee
[URL](https://airflowsummit.org/speakers/jonathan-lee/)

## Jonathan Leek
[URL](https://airflowsummit.org/speakers/jonathan-leek/)

## Jonathan Rainer
[URL](https://airflowsummit.org/speakers/jonathan-rainer/)

## Jonathan Talmi
[URL](https://airflowsummit.org/speakers/jonathan-talmi/)

## Jordan Kail
[URL](https://airflowsummit.org/speakers/jordan-kail/)

## Jorrick Sleijster
[URL](https://airflowsummit.org/speakers/jorrick-sleijster/)

## Jose Puertos
[URL](https://airflowsummit.org/speakers/jose-puertos/)

## Joseph Morotti
[URL](https://airflowsummit.org/speakers/joseph-morotti/)

## Josh Benamram
[URL](https://airflowsummit.org/speakers/josh-benamram/)

## Julian LaNeve
[URL](https://airflowsummit.org/speakers/julian-laneve/)

## Julien Le Dem
[URL](https://airflowsummit.org/speakers/julien-le-dem/)

## Justin Wang
[URL](https://airflowsummit.org/speakers/justin-wang/)

## Jędrzej Matuszak
[URL](https://airflowsummit.org/speakers/jedrzej-matuszak/)

## Kacper Muda
[URL](https://airflowsummit.org/speakers/kacper-muda/)

## Kalyan Reddy
[URL](https://airflowsummit.org/speakers/kalyan-reddy/)

## Kamen Sharlandjiev
[URL](https://airflowsummit.org/speakers/kamen-sharlandjiev/)

## Kamil Bregula
[URL](https://airflowsummit.org/speakers/kamil-bregula/)

## Karan Alang
[URL](https://airflowsummit.org/speakers/karan-alang/)

## Karen Braganza
[URL](https://airflowsummit.org/speakers/karen-braganza/)

## Karolina Rosol
[URL](https://airflowsummit.org/speakers/karolina-rosol/)

## Karthik Dulam
[URL](https://airflowsummit.org/speakers/karthik-dulam/)

## Karthik Kadiyam
[URL](https://airflowsummit.org/speakers/karthik-kadiyam/)

## Karthik Seshadri
[URL](https://airflowsummit.org/speakers/karthik-seshadri/)

## Katarzyna Kałek
[URL](https://airflowsummit.org/speakers/katarzyna-kalek/)

## Kaxil Naik
[URL](https://airflowsummit.org/speakers/kaxil-naik/)

## Ken Obata
[URL](https://airflowsummit.org/speakers/ken-obata/)

## Kengo Seki
[URL](https://airflowsummit.org/speakers/kengo-seki/)

## Kenten Danas
[URL](https://airflowsummit.org/speakers/kenten-danas/)

## Keshav Tyagi
[URL](https://airflowsummit.org/speakers/keshav-tyagi/)

## Kevin Wang
[URL](https://airflowsummit.org/speakers/kevin-wang/)

## Kevin Yang
[URL](https://airflowsummit.org/speakers/kevin-yang/)

## Khadija Al Ahyane
[URL](https://airflowsummit.org/speakers/khadija-al-ahyane/)

## Khaled Hassan
[URL](https://airflowsummit.org/speakers/khaled-hassan/)

## Konrad Schieban
[URL](https://airflowsummit.org/speakers/konrad-schieban/)

## Kowsy Narayan
[URL](https://airflowsummit.org/speakers/kowsy-narayan/)

## Krystal Nzeadibe
[URL](https://airflowsummit.org/speakers/krystal-nzeadibe/)

## Kumuda Sreenivasa
[URL](https://airflowsummit.org/speakers/kumuda-sreenivasa/)

## Kunal Haria
[URL](https://airflowsummit.org/speakers/kunal-haria/)

## Kunal Jain
[URL](https://airflowsummit.org/speakers/kunal-jain/)

## Kuntal Basu
[URL](https://airflowsummit.org/speakers/kuntal-basu/)

## Kushal Thakkar
[URL](https://airflowsummit.org/speakers/kushal-thakkar/)

## Kyle McCluskey
[URL](https://airflowsummit.org/speakers/kyle-mccluskey/)

## Laura Zdanski
[URL](https://airflowsummit.org/speakers/laura-zdanski/)

## Lawrence Gerstley
[URL](https://airflowsummit.org/speakers/lawrence-gerstley/)

## Leah Cole
[URL](https://airflowsummit.org/speakers/leah-cole/)

## Lewis Macdonald
[URL](https://airflowsummit.org/speakers/lewis-macdonald/)

## Lindy Bustabad
[URL](https://airflowsummit.org/speakers/lindy-bustabad/)

## Lisa Cao
[URL](https://airflowsummit.org/speakers/lisa-cao/)

## Luan Moreno Medeiros Maciel
[URL](https://airflowsummit.org/speakers/luan-moreno-medeiros-maciel/)

## Lucas Fonseca
[URL](https://airflowsummit.org/speakers/lucas-fonseca/)

## M. Waqas Shahid
[URL](https://airflowsummit.org/speakers/m-waqas-shahid/)

## Maciej Obuchowski
[URL](https://airflowsummit.org/speakers/maciej-obuchowski/)

## Maciej Oczko
[URL](https://airflowsummit.org/speakers/maciej-oczko/)

## Madhav Khakhar
[URL](https://airflowsummit.org/speakers/madhav-khakhar/)

## Madison Swain-Bowden
[URL](https://airflowsummit.org/speakers/madison-swain-bowden/)

## Maggie Stark
[URL](https://airflowsummit.org/speakers/maggie-stark/)

## Manmeet Kaur
[URL](https://airflowsummit.org/speakers/manmeet-kaur/)

## Marc Lamberti
[URL](https://airflowsummit.org/speakers/marc-lamberti/)

## Marcos Marx
[URL](https://airflowsummit.org/speakers/marcos-marx/)

## Marek Gawiński
[URL](https://airflowsummit.org/speakers/marek-gawinski/)

## Marion Azoulai
[URL](https://airflowsummit.org/speakers/marion-azoulai/)

## Mark Merling
[URL](https://airflowsummit.org/speakers/mark-merling/)

## Marwan Sarieddine
[URL](https://airflowsummit.org/speakers/marwan-sarieddine/)

## Masatake Iwasaki
[URL](https://airflowsummit.org/speakers/masatake-iwasaki/)

## Mateusz Nojek
[URL](https://airflowsummit.org/speakers/mateusz-nojek/)

## Mathieu Monet
[URL](https://airflowsummit.org/speakers/mathieu-monet/)

## Matt Koski
[URL](https://airflowsummit.org/speakers/matt-koski/)

## Matt McCormack
[URL](https://airflowsummit.org/speakers/matt-mccormack/)

## Matthew Davis
[URL](https://airflowsummit.org/speakers/matthew-davis/)

## Matthew Stavinga
[URL](https://airflowsummit.org/speakers/matthew-stavinga/)

## Mauricio De Diana
[URL](https://airflowsummit.org/speakers/mauricio-de-diana/)

## Max Payton
[URL](https://airflowsummit.org/speakers/max-payton/)

## Maxime Beauchemin
[URL](https://airflowsummit.org/speakers/maxime-beauchemin/)

## Mayank Chopra
[URL](https://airflowsummit.org/speakers/mayank-chopra/)

## Megan Parker
[URL](https://airflowsummit.org/speakers/megan-parker/)

## Meni Shmueli
[URL](https://airflowsummit.org/speakers/meni-shmueli/)

## Michael Atondo
[URL](https://airflowsummit.org/speakers/michael-atondo/)

## Michael Collado
[URL](https://airflowsummit.org/speakers/michael-collado/)

## Michael Hewitt
[URL](https://airflowsummit.org/speakers/michael-hewitt/)

## Michael Juster
[URL](https://airflowsummit.org/speakers/michael-juster/)

## Michael Kotliar
[URL](https://airflowsummit.org/speakers/michael-kotliar/)

## Michael Robinson
[URL](https://airflowsummit.org/speakers/michael-robinson/)

## Michael Winser
[URL](https://airflowsummit.org/speakers/michael-winser/)

## Michal Dura
[URL](https://airflowsummit.org/speakers/michal-dura/)

## Michał Modras
[URL](https://airflowsummit.org/speakers/michal-modras/)

## Michel Tricot
[URL](https://airflowsummit.org/speakers/michel-tricot/)

## Michelle Winters
[URL](https://airflowsummit.org/speakers/michelle-winters/)

## Mihail Petkov
[URL](https://airflowsummit.org/speakers/mihail-petkov/)

## Mikaela Pisani
[URL](https://airflowsummit.org/speakers/mikaela-pisani/)

## Mike Ellis
[URL](https://airflowsummit.org/speakers/mike-ellis/)

## Mike Hirsch
[URL](https://airflowsummit.org/speakers/mike-hirsch/)

## Milcah Mbithi
[URL](https://airflowsummit.org/speakers/milcah-mbithi/)

## Mili Tripathi
[URL](https://airflowsummit.org/speakers/mili-tripathi/)

## Minh Nguyen
[URL](https://airflowsummit.org/speakers/minh-nguyen/)

## Miquel Angel Andreu Febrer
[URL](https://airflowsummit.org/speakers/miquel-angel-andreu-febrer/)

## Mocheng Guo
[URL](https://airflowsummit.org/speakers/mocheng-guo/)

## Mohammed Marragh
[URL](https://airflowsummit.org/speakers/mohammed-marragh/)

## Moulay Zaidane Al Bahi Draidia
[URL](https://airflowsummit.org/speakers/moulay-zaidane-al-bahi-draidia/)

## Muhammed Irshad
[URL](https://airflowsummit.org/speakers/muhammed-irshad/)

## Multiple Speakers
[URL](https://airflowsummit.org/speakers/multiple-speakers/)

## Naga Durga Rao Dindi
[URL](https://airflowsummit.org/speakers/naga-durga-rao-dindi/)

## Nanxi Chen
[URL](https://airflowsummit.org/speakers/nanxi-chen/)

## Naresh Yegireddi
[URL](https://airflowsummit.org/speakers/naresh-yegireddi/)

## Naseem Shah
[URL](https://airflowsummit.org/speakers/naseem-shah/)

## Nathan Hadfield
[URL](https://airflowsummit.org/speakers/nathan-hadfield/)

## Nathaniel Rose
[URL](https://airflowsummit.org/speakers/nathaniel-rose/)

## Navid Aghdaie
[URL](https://airflowsummit.org/speakers/navid-aghdaie/)

## Nawfel Bacha
[URL](https://airflowsummit.org/speakers/nawfel-bacha/)

## Neha Singla
[URL](https://airflowsummit.org/speakers/neha-singla/)

## Nehil Jain
[URL](https://airflowsummit.org/speakers/nehil-jain/)

## Nicholas Pataki
[URL](https://airflowsummit.org/speakers/nicholas-pataki/)

## Nicholas Redd
[URL](https://airflowsummit.org/speakers/nicholas-redd/)

## Nick Acosta
[URL](https://airflowsummit.org/speakers/nick-acosta/)

## Nick Bilozerov
[URL](https://airflowsummit.org/speakers/nick-bilozerov/)

## Nick Luckey
[URL](https://airflowsummit.org/speakers/nick-luckey/)

## Nikhil Nandoskar
[URL](https://airflowsummit.org/speakers/nikhil-nandoskar/)

## Niko Oliveira
[URL](https://airflowsummit.org/speakers/niko-oliveira/)

## Nitish Victor
[URL](https://airflowsummit.org/speakers/nitish-victor/)

## Noam Elfanbaum
[URL](https://airflowsummit.org/speakers/noam-elfanbaum/)

## Ole Christian Langfjæran
[URL](https://airflowsummit.org/speakers/ole-christian-langfjaeran/)

## Oleksandr Slynko
[URL](https://airflowsummit.org/speakers/oleksandr-slynko/)

## Oliver Dykstra
[URL](https://airflowsummit.org/speakers/oliver-dykstra/)

## Olivier Daneau
[URL](https://airflowsummit.org/speakers/olivier-daneau/)

## Oluwafemi Olawoyin
[URL](https://airflowsummit.org/speakers/oluwafemi-olawoyin/)

## Ori Peri
[URL](https://airflowsummit.org/speakers/ori-peri/)

## Oscar Ligthart
[URL](https://airflowsummit.org/speakers/oscar-ligthart/)

## Ozcan Ilikhan
[URL](https://airflowsummit.org/speakers/ozcan-ilikhan/)

## Palak Patel
[URL](https://airflowsummit.org/speakers/palak-patel/)

## Palanieppan Muthiah
[URL](https://airflowsummit.org/speakers/palanieppan-muthiah/)

## Pankaj Koti
[URL](https://airflowsummit.org/speakers/pankaj-koti/)

## Pankaj Singh
[URL](https://airflowsummit.org/speakers/pankaj-singh/)

## Parnab Basak
[URL](https://airflowsummit.org/speakers/parnab-basak/)

## Patricio Garza
[URL](https://airflowsummit.org/speakers/patricio-garza/)

## Paul Balm
[URL](https://airflowsummit.org/speakers/paul-balm/)

## Paul Dittamo
[URL](https://airflowsummit.org/speakers/paul-dittamo/)

## Pavan kumar Gopidesu
[URL](https://airflowsummit.org/speakers/pavan-kumar-gopidesu/)

## Pavan Kumar Naikal
[URL](https://airflowsummit.org/speakers/pavan-kumar-naikal/)

## Pawel Leszczynski
[URL](https://airflowsummit.org/speakers/pawel-leszczynski/)

## Peeyush Rai
[URL](https://airflowsummit.org/speakers/peeyush-rai/)

## Pei-Chi (Miko) Chen
[URL](https://airflowsummit.org/speakers/pei-chi-miko-chen/)

## Peiqiu Tian
[URL](https://airflowsummit.org/speakers/peiqiu-tian/)

## Pete DeJoy
[URL](https://airflowsummit.org/speakers/pete-dejoy/)

## Phani Kumar
[URL](https://airflowsummit.org/speakers/phani-kumar/)

## Philipp Lang
[URL](https://airflowsummit.org/speakers/philipp-lang/)

## Philippe Gagnon
[URL](https://airflowsummit.org/speakers/philippe-gagnon/)

## Pierre Jeambrun
[URL](https://airflowsummit.org/speakers/pierre-jeambrun/)

## Ping Zhang
[URL](https://airflowsummit.org/speakers/ping-zhang/)

## Piotr Dziuba
[URL](https://airflowsummit.org/speakers/piotr-dziuba/)

## Piotr Leśniak
[URL](https://airflowsummit.org/speakers/piotr-lesniak/)

## Piotr Wieczorek
[URL](https://airflowsummit.org/speakers/piotr-wieczorek/)

## Plinio Guzman
[URL](https://airflowsummit.org/speakers/plinio-guzman/)

## Pooja Pal
[URL](https://airflowsummit.org/speakers/pooja-pal/)

## Prakash Nandha Mukunthan
[URL](https://airflowsummit.org/speakers/prakash-nandha-mukunthan/)

## Prateek Chawla
[URL](https://airflowsummit.org/speakers/prateek-chawla/)

## Preethi Ganeshan
[URL](https://airflowsummit.org/speakers/preethi-ganeshan/)

## Prekshi Vyas
[URL](https://airflowsummit.org/speakers/prekshi-vyas/)

## Prince Bose
[URL](https://airflowsummit.org/speakers/prince-bose/)

## Priyanka Samanta
[URL](https://airflowsummit.org/speakers/priyanka-samanta/)

## Przemek Więch
[URL](https://airflowsummit.org/speakers/przemek-wiech/)

## Purshotam Shah
[URL](https://airflowsummit.org/speakers/purshotam-shah/)

## Pádraic Slattery
[URL](https://airflowsummit.org/speakers/padraic-slattery/)

## QP Hou
[URL](https://airflowsummit.org/speakers/qp-hou/)

## Rachael Deacon-Smith
[URL](https://airflowsummit.org/speakers/rachael-deacon-smith/)

## Rachana Mahapatra
[URL](https://airflowsummit.org/speakers/rachana-mahapatra/)

## Rachel Sun
[URL](https://airflowsummit.org/speakers/rachel-sun/)

## Rafael Pierre
[URL](https://airflowsummit.org/speakers/rafael-pierre/)

## Rafael Ribaldo
[URL](https://airflowsummit.org/speakers/rafael-ribaldo/)

## Rafal Biegacz
[URL](https://airflowsummit.org/speakers/rafal-biegacz/)

## Rafay Aleem
[URL](https://airflowsummit.org/speakers/rafay-aleem/)

## Rahul Gade
[URL](https://airflowsummit.org/speakers/rahul-gade/)

## Rahul Vats
[URL](https://airflowsummit.org/speakers/rahul-vats/)

## Raj Katakam
[URL](https://airflowsummit.org/speakers/raj-katakam/)

## Raj Ramalingam
[URL](https://airflowsummit.org/speakers/raj-ramalingam/)

## Rajesh Bishundeo
[URL](https://airflowsummit.org/speakers/rajesh-bishundeo/)

## Rajesh Gundugollu
[URL](https://airflowsummit.org/speakers/rajesh-gundugollu/)

## Rakesh Kumar Tai
[URL](https://airflowsummit.org/speakers/rakesh-kumar-tai/)

## Ramajayam Gopithirumal
[URL](https://airflowsummit.org/speakers/ramajayam-gopithirumal/)

## Ramesh Babu
[URL](https://airflowsummit.org/speakers/ramesh-babu/)

## Ramit Kataria
[URL](https://airflowsummit.org/speakers/ramit-kataria/)

## Ramya  Pappu
[URL](https://airflowsummit.org/speakers/ramya-pappu/)

## Raphaël Vandon
[URL](https://airflowsummit.org/speakers/raphael-vandon/)

## Ravi Autar
[URL](https://airflowsummit.org/speakers/ravi-autar/)

## Ray Carroll
[URL](https://airflowsummit.org/speakers/ray-carroll/)

## Ricardo Sueiras
[URL](https://airflowsummit.org/speakers/ricardo-sueiras/)

## Rich Bowen
[URL](https://airflowsummit.org/speakers/rich-bowen/)

## Rishi Kar
[URL](https://airflowsummit.org/speakers/rishi-kar/)

## Ritika Jain
[URL](https://airflowsummit.org/speakers/ritika-jain/)

## Roberto Santamaria
[URL](https://airflowsummit.org/speakers/roberto-santamaria/)

## Rodrigo Loredo
[URL](https://airflowsummit.org/speakers/rodrigo-loredo/)

## Roi Teveth
[URL](https://airflowsummit.org/speakers/roi-teveth/)

## Roman Khomenko
[URL](https://airflowsummit.org/speakers/roman-khomenko/)

## Ross Turk
[URL](https://airflowsummit.org/speakers/ross-turk/)

## Roy Noyman
[URL](https://airflowsummit.org/speakers/roy-noyman/)

## Rumeysa Ozaydin
[URL](https://airflowsummit.org/speakers/rumeysa-ozaydin/)

## Russell Dervay
[URL](https://airflowsummit.org/speakers/russell-dervay/)

## Russell Lamb
[URL](https://airflowsummit.org/speakers/russell-lamb/)

## Ry Walker
[URL](https://airflowsummit.org/speakers/ry-walker/)

## Ryan Delgado
[URL](https://airflowsummit.org/speakers/ryan-delgado/)

## Ryan Hatter
[URL](https://airflowsummit.org/speakers/ryan-hatter/)

## Ryan Stevens
[URL](https://airflowsummit.org/speakers/ryan-stevens/)

## Sabrina Liu
[URL](https://airflowsummit.org/speakers/sabrina-liu/)

## Sagar Sharma
[URL](https://airflowsummit.org/speakers/sagar-sharma/)

## Salih Goktug Kose
[URL](https://airflowsummit.org/speakers/salih-goktug-kose/)

## Sam Bail
[URL](https://airflowsummit.org/speakers/sam-bail/)

## Sam Dengler
[URL](https://airflowsummit.org/speakers/sam-dengler/)

## Sam Wheating
[URL](https://airflowsummit.org/speakers/sam-wheating/)

## Sanchit Sreekanth
[URL](https://airflowsummit.org/speakers/sanchit-sreekanth/)

## Sandeep Bommisetti
[URL](https://airflowsummit.org/speakers/sandeep-bommisetti/)

## Sarah Johnson
[URL](https://airflowsummit.org/speakers/sarah-johnson/)

## Sarah Krasnik
[URL](https://airflowsummit.org/speakers/sarah-krasnik/)

## Sathish Kumar Thangaraj
[URL](https://airflowsummit.org/speakers/sathish-kumar-thangaraj/)

## Satya Tej
[URL](https://airflowsummit.org/speakers/satya-tej/)

## Saurabh Gupta
[URL](https://airflowsummit.org/speakers/saurabh-gupta/)

## Savin Goyal
[URL](https://airflowsummit.org/speakers/savin-goyal/)

## Sean Bjurstrom
[URL](https://airflowsummit.org/speakers/sean-bjurstrom/)

## Sean Ghaeli
[URL](https://airflowsummit.org/speakers/sean-ghaeli/)

## Sean Lewis
[URL](https://airflowsummit.org/speakers/sean-lewis/)

## Sébastien Crocquevieille
[URL](https://airflowsummit.org/speakers/sebastien-crocquevieille/)

## Sergio Camilo Fandiño Hernández
[URL](https://airflowsummit.org/speakers/sergio-camilo-fandino-hernandez/)

## Shahar Epstein
[URL](https://airflowsummit.org/speakers/shahar-epstein/)

## Shalabh Agarwal
[URL](https://airflowsummit.org/speakers/shalabh-agarwal/)

## Shanelle Roman
[URL](https://airflowsummit.org/speakers/shanelle-roman/)

## Shirshanka Das
[URL](https://airflowsummit.org/speakers/shirshanka-das/)

## Shivam Rastogi
[URL](https://airflowsummit.org/speakers/shivam-rastogi/)

## Shivnath Babu
[URL](https://airflowsummit.org/speakers/shivnath-babu/)

## Shobhit Shah
[URL](https://airflowsummit.org/speakers/shobhit-shah/)

## Shoubhik Bose
[URL](https://airflowsummit.org/speakers/shoubhik-bose/)

## Shrividya Hegde
[URL](https://airflowsummit.org/speakers/shrividya-hegde/)

## Shubham Mehta
[URL](https://airflowsummit.org/speakers/shubham-mehta/)

## Shubham Raj
[URL](https://airflowsummit.org/speakers/shubham-raj/)

## Silver Pang
[URL](https://airflowsummit.org/speakers/silver-pang/)

## Siraj Malik
[URL](https://airflowsummit.org/speakers/siraj-malik/)

## Sneha Rao
[URL](https://airflowsummit.org/speakers/sneha-rao/)

## Snir Israeli
[URL](https://airflowsummit.org/speakers/snir-israeli/)

## Sophie Keith
[URL](https://airflowsummit.org/speakers/sophie-keith/)

## Soren Archibald
[URL](https://airflowsummit.org/speakers/soren-archibald/)

## Spencer Tollefson
[URL](https://airflowsummit.org/speakers/spencer-tollefson/)

## Spencer Weeks
[URL](https://airflowsummit.org/speakers/spencer-weeks/)

## Srabasti Banerjee
[URL](https://airflowsummit.org/speakers/srabasti-banerjee/)

## Sriharsh Adari
[URL](https://airflowsummit.org/speakers/sriharsh-adari/)

## Srini Kadamati
[URL](https://airflowsummit.org/speakers/srini-kadamati/)

## Sriram Ramarathnam
[URL](https://airflowsummit.org/speakers/sriram-ramarathnam/)

## Sriram Vamsi Ilapakurthy
[URL](https://airflowsummit.org/speakers/sriram-vamsi-ilapakurthy/)

## Stanislav Repka
[URL](https://airflowsummit.org/speakers/stanislav-repka/)

## Stanisław Smyl
[URL](https://airflowsummit.org/speakers/stanislaw-smyl/)

## Stas Bytsko
[URL](https://airflowsummit.org/speakers/stas-bytsko/)

## Stephanie Niu
[URL](https://airflowsummit.org/speakers/stephanie-niu/)

## Steve Sawyer
[URL](https://airflowsummit.org/speakers/steve-sawyer/)

## Steven Woods
[URL](https://airflowsummit.org/speakers/steven-woods/)

## Stoney Gwitira
[URL](https://airflowsummit.org/speakers/stoney-gwitira/)

## Suba Palanisamy
[URL](https://airflowsummit.org/speakers/suba-palanisamy/)

## Subash Canapathy
[URL](https://airflowsummit.org/speakers/subash-canapathy/)

## Subramanian Vellaiyan
[URL](https://airflowsummit.org/speakers/subramanian-vellaiyan/)

## Sudarshan Chaudhari
[URL](https://airflowsummit.org/speakers/sudarshan-chaudhari/)

## Sumit Maheshwari
[URL](https://airflowsummit.org/speakers/sumit-maheshwari/)

## Sung Yun
[URL](https://airflowsummit.org/speakers/sung-yun/)

## Sungji Yang
[URL](https://airflowsummit.org/speakers/sungji-yang/)

## Sushmita Barthakur
[URL](https://airflowsummit.org/speakers/sushmita-barthakur/)

## Syed Hussain
[URL](https://airflowsummit.org/speakers/syed-hussain/)

## Tahir Fayyaz
[URL](https://airflowsummit.org/speakers/tahir-fayyaz/)

## Tala Karadsheh
[URL](https://airflowsummit.org/speakers/tala-karadsheh/)

## Tamás Németh
[URL](https://airflowsummit.org/speakers/tamas-nemeth/)

## Tao Feng
[URL](https://airflowsummit.org/speakers/tao-feng/)

## Tarasov Alexey
[URL](https://airflowsummit.org/speakers/tarasov-alexey/)

## Tatiana Al-Chueyr Martins
[URL](https://airflowsummit.org/speakers/tatiana-al-chueyr-martins/)

## Taylor Facen
[URL](https://airflowsummit.org/speakers/taylor-facen/)

## Theo Lebrun
[URL](https://airflowsummit.org/speakers/theo-lebrun/)

## Thomas Tauber-Marshall
[URL](https://airflowsummit.org/speakers/thomas-tauber-marshall/)

## Tomasz Urbaszek
[URL](https://airflowsummit.org/speakers/tomasz-urbaszek/)

## Traey Hatch
[URL](https://airflowsummit.org/speakers/traey-hatch/)

## Trevor DeVore
[URL](https://airflowsummit.org/speakers/trevor-devore/)

## Tzu-ping Chung
[URL](https://airflowsummit.org/speakers/tzu-ping-chung/)

## Udit Saxena
[URL](https://airflowsummit.org/speakers/udit-saxena/)

## Uma Ramadoss
[URL](https://airflowsummit.org/speakers/uma-ramadoss/)

## Uriel Muñoz
[URL](https://airflowsummit.org/speakers/uriel-munoz/)

## Utkarsh Sharma
[URL](https://airflowsummit.org/speakers/utkarsh-sharma/)

## Vanessa Sochat
[URL](https://airflowsummit.org/speakers/vanessa-sochat/)

## Vara Prasad Regani
[URL](https://airflowsummit.org/speakers/vara-prasad-regani/)

## Varun Srinivas
[URL](https://airflowsummit.org/speakers/varun-srinivas/)

## Vasantha Kosuri Marshall
[URL](https://airflowsummit.org/speakers/vasantha-kosuri-marshall/)

## Venkata Jagannath
[URL](https://airflowsummit.org/speakers/venkata-jagannath/)

## Vicente Rubén Del Pino Ruiz
[URL](https://airflowsummit.org/speakers/vicente-ruben-del-pino-ruiz/)

## Victor Aoqui
[URL](https://airflowsummit.org/speakers/victor-aoqui/)

## Victor Chiapaikeo
[URL](https://airflowsummit.org/speakers/victor-chiapaikeo/)

## Victor Shafran
[URL](https://airflowsummit.org/speakers/victor-shafran/)

## Victoria Varney
[URL](https://airflowsummit.org/speakers/victoria-varney/)

## Vikram Koka
[URL](https://airflowsummit.org/speakers/vikram-koka/)

## Vincent Beck
[URL](https://airflowsummit.org/speakers/vincent-beck/)

## Vincent La
[URL](https://airflowsummit.org/speakers/vincent-la/)

## Vinod Jayendra
[URL](https://airflowsummit.org/speakers/vinod-jayendra/)

## Vinoo Ganesh
[URL](https://airflowsummit.org/speakers/vinoo-ganesh/)

## Vipin Kataria
[URL](https://airflowsummit.org/speakers/vipin-kataria/)

## Viraj Parekh
[URL](https://airflowsummit.org/speakers/viraj-parekh/)

## Vishal Vijayvargiya
[URL](https://airflowsummit.org/speakers/vishal-vijayvargiya/)

## Vishvesh Pandey
[URL](https://airflowsummit.org/speakers/vishvesh-pandey/)

## Vladi Nekolov
[URL](https://airflowsummit.org/speakers/vladi-nekolov/)

## Vladislav Repev
[URL](https://airflowsummit.org/speakers/vladislav-repev/)

## Volker Janz
[URL](https://airflowsummit.org/speakers/volker-janz/)

## Wanda Kinasih
[URL](https://airflowsummit.org/speakers/wanda-kinasih/)

## Wei Lee
[URL](https://airflowsummit.org/speakers/wei-lee/)

## Wensi Hu
[URL](https://airflowsummit.org/speakers/wensi-hu/)

## William Orgertrice
[URL](https://airflowsummit.org/speakers/william-orgertrice/)

## Willy Lulciuc
[URL](https://airflowsummit.org/speakers/willy-lulciuc/)

## Wuchao Chen
[URL](https://airflowsummit.org/speakers/wuchao-chen/)

## Wyatt Shapiro
[URL](https://airflowsummit.org/speakers/wyatt-shapiro/)

## Xiaodong Deng
[URL](https://airflowsummit.org/speakers/xiaodong-deng/)

## Xiaoqin Zhu
[URL](https://airflowsummit.org/speakers/xiaoqin-zhu/)

## Xiongfeng Song
[URL](https://airflowsummit.org/speakers/xiongfeng-song/)

## Xu Han
[URL](https://airflowsummit.org/speakers/xu-han/)

## Yarden Wolf
[URL](https://airflowsummit.org/speakers/yarden-wolf/)

## Yashas Prashanth
[URL](https://airflowsummit.org/speakers/yashas-prashanth/)

## Yetunde Dada
[URL](https://airflowsummit.org/speakers/yetunde-dada/)

## Yifan (Stefan) Wang
[URL](https://airflowsummit.org/speakers/yifan-wang/)

## Yingbo Wang
[URL](https://airflowsummit.org/speakers/yingbo-wang/)

## Yu Lung Law
[URL](https://airflowsummit.org/speakers/yu-lung-law/)

## Yuan Luo
[URL](https://airflowsummit.org/speakers/yuan-luo/)

## Yuanmeng Zeng
[URL](https://airflowsummit.org/speakers/yuanmeng-zeng/)

## Yuhang Huang
[URL](https://airflowsummit.org/speakers/yuhang-huang/)

## Yulei Li
[URL](https://airflowsummit.org/speakers/yulei-li/)

## Yunhao Qing
[URL](https://airflowsummit.org/speakers/yunhao-qing/)

## Yuri Desyatnik
[URL](https://airflowsummit.org/speakers/yuri-desyatnik/)

## Yuvaraj Sankaran
[URL](https://airflowsummit.org/speakers/yuvaraj-sankaran/)

## Zach Gottesman
[URL](https://airflowsummit.org/speakers/zach-gottesman/)

## Zachary Bannor
[URL](https://airflowsummit.org/speakers/zachary-bannor/)

## Zdravko Hvarlingov
[URL](https://airflowsummit.org/speakers/zdravko-hvarlingov/)

## Zhang Zhang
[URL](https://airflowsummit.org/speakers/zhang-zhang/)

## Zhe You Liu
[URL](https://airflowsummit.org/speakers/zhe-you-liu/)

## Zhengyi Liu
[URL](https://airflowsummit.org/speakers/zhengyi-liu/)

## Zohar Donenhirsh
[URL](https://airflowsummit.org/speakers/zohar-donenhirsh/)