NCR Voyix Retail Analytics AI team offers ML products for retailers while embracing Airflow as its MLOps Platform. As the team is small and there have been twice as many data scientists as engineers, we encountered challenges in making Airflow accessible to the scientists:
As they come from diverse programming backgrounds, we needed an architecture enabling them to develop production-ready ML workflows without prior knowledge of Airflow.
Due to dynamic product demands, we had to implement a mechanism to interchange Airflow operators effortlessly.
As workflows serve multiple customers, they should be easily configurable and simultaneously deployable.
We came up with the following architecture to deal with the above:
Enabling our data scientists to formulate ML workflows as structured Python files.
Seamlessly converting the workflows into Airflow DAGs while aggregating their steps to be executed on different Airflow operators.
Deploying DAGs via CI/CD’s UI to the DAGs folder for all customers while considering definitions for each in their configuration files.
In this session, we will cover Airflow’s evolution in our team and review the concepts of our architecture.
