Airflow at Faire: Democratizing ML feature store framework at scale

Data science and machine learning are at the heart of Faire’s industry-celebrated marketplace (a16z top-ranked marketplace) and drive powerful search, navigation, and risk functions which are powered by ML models that are trained on 3000+ features defined by our data scientists.

Previously, defining, backfilling and maintaining feature lifecycle was error-prone. Having a framework built on top of Airflow has empowered them to maintain and deploy their changes independently.

We will explore:

• How to leverage Airflow as a tool that can power ML training and extend it with a framework that powers feature store.
• Enabling data scientists to define new features and backfill them (common problem in the ML world) using dynamic DAGs.

The talk will provide valuable insights into how Faire constructed a framework that builds datasets to train models. Plus, how empowering end-users with tools isn’t something to fear but frees up engineering teams to focus on strategic initiatives.