Speculative Testing at Google with Transition Prediction

Google's approach to testing includes both testing prior to code submission (for fast validation) and after code submission (for comprehensive validation). However, Google's ever growing testing demand has lead to increased continuous integration cycle latency and machine costs. When the post code submission continuous integration cycles get longer, it delays detecting breakages in the main repository which increases developer friction and lowers productivity. To mitigate this without increasing resource demand, Google is implementing Postsubmit Speculative Cycles in their Test Automation Platform (TAP). Speculative Cycles prioritize finding novel breakages faster. In this paper we present our new test scheduling architecture and the machine learning system (Transition Prediction) driving it. Both the ML system and the end-to-end test scheduling system are empirically evaluated on 3-months of our production data (120 billion test × cycle pairs, 7.7 million breaking targets, with $\sim20$ thousand unique breakages). Using Speculative Cycles we observed a median (p50) reduction of approximately 65% (from 107 to 37 minutes) in the time taken to detect novel breaking targets.