The Messy Middle: When Your Data Team Doesn't Need a Streaming Engine

There’s a class of workload that doesn’t belong in your streaming stack. A team needs to react to data arriving in S3 or a message landing in Kafka. The SLA is minutes. Someone reaches for Flink because the orchestrator can’t trigger on events. Six months later, you’re running a streaming app for what is a bounded computation with a latency requirement.

This talk names that pattern, the “messy middle,” and argues that Airflow 3 eliminates the gap that pushed these workloads to streaming. Asset Watchers monitor external sources through async triggers, firing DAGs within minutes of event arrival. Assets turn data products into scheduling primitives. Partitions let Airflow reason about which slices of a dataset are ready.

Airflow is still a batch orchestrator. It won’t replace Flink for sub-minute guarantees, stateful processing, or windowed aggregations. I’ll be direct about the boundaries and present a framework for when streaming is actually the right call.

Attendees will leave with a precise definition of the messy middle, event-driven orchestration patterns for Airflow 3, and a way to evaluate orchestration vs. streaming on operational cost, not just latency.