Automotive A/B testing: Challenges and Lessons Learned from Practice

Over the past 15 years, A/B testing has been a critical tool for accurate prioritization of development efforts in online and web-facing companies. As automotive companies progress on their digitalization process, A/B testing and other experimentation techniques start to be adopted. However, specific characteristics of the automotive software industry create additional challenges to the successful adoption of A/B testing. Recently, research has been conducted to investigate the challenges and opportunities for experimentation techniques in the automotive and more generally in the embedded systems domain. However, despite the collaboration with industry, previous research was based on either hypothesized or toy scenarios in companies seeking, but not yet running experimentation. Utilizing a case study method, we investigate the challenges of adopting A/B testing in two large-scale automotive companies that are currently running or preparing for their first A/B testing. The contribution of this paper is two-fold. First, we present our main findings in terms of the challenges of real A/B testing iterations in automotive vehicles. Second, we present the current, potential solutions and lessons learned from applying A/B testing in the automotive domain.