Evolving Software to be ML-Driven Utilizing Real-World A/B Testing: Experiences, Insights, Challenges

ML-driven software is heralded as the next majoradvancement in software engineering; existing software todaycan benefit from being evolved to be ML-driven. In this paper, we contribute practical knowledge about evolving software tobe ML-driven, utilizing real-world A/B testing. We draw onexperiences evolving two software features from the Windowsoperating system to be ML-driven, with more than ten real-world A/B tests on millions of PCs over more than two years.We discuss practical reasons for using A/B testing to engineerML-driven software, insights for success, as well as on-going real-world challenges. This knowledge may help practitioners, as wellas help direct future research and innovations.