Building on Cosmos: Making dbt on Airflow Easy

Speaker(s):
Hugo Hobson
Hugo Hobson
By Hugo Hobson

Balyasny Asset Management (BAM) is a diversified global investment firm founded in 2001 with over $20 billion in assets under management.

As dbt took hold at BAM, we had multiple teams building dbt projects against Snowflake, Redshift, and SQL Server. The common question was: How can we quickly and easily productionise our projects?

Airflow is the orchestrator of choice at BAM, but our dbt users ranged from Airflow power users to people who’d never heard of Airflow before. We built a single solution on top of Cosmos that allowed us to:

  • Decouple the dbt project from the Airflow repository
  • Have each dbt node run as a separate Airflow task
  • Allow users to run dbt with little to no Airflow knowledge
  • Enable users to have fine-grained control over how dbt is run and to combine it with other Airflow tasks
  • Provide observability, monitoring, and alerting