Exploring the Triggering Modes of Spectrum-Based Fault Localization: An Industrial Case

Fault localization is important for software development and maintenance. Among existing techniques, spectrum-based fault localization (SBFL) is effective to locate bugs based on the execution coverage of passed and failed tests. However, current SBFL tools require the execution of all tests before suggesting any ranked list of suspicious locations. In reality, such all-test execution can be very time-consuming; SBFL's outputs based on the whole-suite execution can significantly delay developers' debugging activities and jeopardize their productivity. For this paper, we were curious whether we can apply SBFL immediately after seeing one or several test failures, instead of waiting for all tests to finish their run. Specifically, with 28 injected bugs and 13 real bugs in a close-sourced software product, we collected the statement-level coverage for each test case, and investigated the usage of 25 alternative SBFL formulas. We triggered SBFL in five modes: (i) after the first test failure, (ii) after the first failure and some extra passed tests, (iii) after every test failure, (iv) at a specified time interval (e.g., every 2 minutes), or (v) after the complete execution of all tests. Our study shows interesting results. Compared with whole-suite execution, triggering SBFL formulas earlier based on partial execution helps locate bugs more effectively. Among the five modes, the first-failure-driven mode works best. Additionally, we conducted similar experiments on 57 real bugs from the Defects4J dataset and observed similar phenomena. Our observations imply that instead of waiting for the completion of all test runs, it is quite promising to apply SBFL formulas immediately after the initial test failure. In this way, developers are likely to get better suggestions within a shorter period of time. Our research will help developers better adopt SBFL in practice.