Autonomous, Not Unsupervised: Agent-Authored Pipelines on Airflow at Together AI

At Together AI, AI agents have become the primary authors of our production data pipelines — and Airflow is what makes that safe to do. Agents do the building. Airflow gives us the surface to set the rules, see what’s happening, and step in when we need to. The interesting part is what each side has to look like for that to actually work in production. This talk is a field report on that relationship. We’ll walk how we got from a world where humans wrote SQL by hand and dashboards refreshed nightly to one where agents make hundreds of queries per session, catalog thousands of tables across engines, and ship pipelines in hours instead of weeks. The platform now spans twelve dbt projects across billing, inference, and analytics — all of it agent-authored, all of it running through Airflow.

We’ll cover the conventions that make agent-authored pipelines legible, the guardrails that make them safe, and the runtime patterns that make them recoverable. And we’ll dig into why oversight built for human authors breaks down the moment your author is non-deterministic — and what it takes to rebuild that oversight as something agents and humans can share.

A short live demo closes the loop: a single ticket goes in, a production-ready pipeline comes out, and the platform stays in control on both sides.

Attendees will leave with adoption patterns for agent-authored pipelines, a model for oversight that scales with non-deterministic authors, and the conventions-as-code playbook we use to keep agents productive without losing control.