Software engineering was built on the assumption that engineers own their inputs. Data engineering broke that assumption years ago. Data engineers own the DAG, but the schema, the arrival time, the volume, and the meaning of the data all get decided upstream, by teams who have never read the code. Every data engineer knows this but putting numbers on it is harder.
I looked at 400 million production pipeline runs on Astro, our managed Airflow service, across 2025. The failure rate in the 24 hours before a deploy is identical to the 24 hours after. The boundary everyone watches is not where things break. Most of the effort we spend on data reliability is aimed at the wrong layer.
Agentic AI is inheriting the same problem with a twist: The model itself is also a non-deterministic dependency you don’t control. Software engineers are rediscovering input-space reliability and renaming it as they go. Evals are data quality tests, model drift is schema drift, and guardrails are the recovery layer.
Data engineers have run this discipline for a decade, and it’s what agentic orchestration needs next. I’ll show you where pipeline failures actually come from, why the deploy dashboard misleads, and how to make the case to whoever owns the roadmap. The reliability discipline the AI push needs is already on your team.
Constance Martineau
Staff Product Manager | Astronomer