Threats to Validity in Experimenting Mutation-Based Fault Localization

Mutation-based fault localization (MBFL) is a promising direction toward improving fault localization accuracy by leveraging dynamic information extracted by mutation testing on a target program. One issue in investigating MBFL techniques is that experimental evaluations are prone to various validity threats because there are many factors to control in generating and running mutants for fault localization. To understand different validity threats in experimenting MBFL techniques, this paper reports our re-production of the MBFL assessments with Defects4J originally studied by Pearson et al. Having the JFreeChart artifacts of Defects4J (total 26 bug cases) as study objects, we conducted the same empirical evaluation on two MBFL (Metallaxis and MUSE) and six SBFL techniques (DStar, Op2, Ochiai, Jaccard, Barniel, Tarantula) while identifying and managing validity threats in alternative ways. As results, we found that the evaluation on the studied techniques change in many parts from the original results, thus, the identified validity threats should be managed carefully at designing and conducting empirical assessments on MBFL techniques.