Twitch, the world’s leading live streaming platform, has a massive user base of over 140 million active users and an incredibly complex recommendation system to deliver a personalized and engaging experience to its users.
In this talk, we will dive into how Twitch leverages the power of Apache Airflow to manage and orchestrate the training and deployment of its recommendation models. You will learn about the scale of Twitch’s reach and the challenges we faced in building a scalable, reliable, and developer-friendly recommendation system.
We will also highlight the custom tooling built internally to make it easier for Twitch’s applied scientists to iterate and develop confidently with Airflow. These customizations have helped Twitch streamline its processes, control costs, improve collaboration between teams, and ensure a seamless experience for internal users of Airflow.