Exploring Data Cleanness in Defects4J and Its Influence on Fault Localization Efficiency

Defects4J stands out as the most popular benchmark in software testing research, known for its comprehensive collection of real bugs from open-source systems. This paper presents an in-depth study of Defects4J's fault-triggering tests, particularly examining the influence of developer modifications post-bug reports on spectrum-based fault localization (SBFL) techniques. Our findings reveal that 55% of these tests were newly added and 22% modified with developer knowledge, impacting the accuracy of SBFL. Notably, SBFL techniques' performance drops significantly (up to -415% in Mean First Rank) when developer knowledge is absent in the tests. We provide a curated dataset of bugs without this knowledge, facilitating more realistic evaluations of SBFL techniques using Defects4J. This research offers insights for the development of future bug benchmarks.