Presubmit Rescue: Automatically Ignoring Flaky Test Executions

Flaky tests often give misleading signals and slow down the speed at which Software Engineers can develop. Traditional ways to help mitigate the developer time loss revolve around identifying tests which are flaky and choosing not to run them or adding additional attempts to reduce the chance of them flakily failing. We present Google’s newest approach to the problem that tries to maintain the usefulness of flaky tests without being as time and resource wasteful as executing additional test attempts (deflaking). Presubmit Rescue is a machine learning model that predicts if a failed test execution was caused by a flake. Predicted flaky failures can be automatically ignored to save developer time without spending machine time for retries or developer time on debugging.