Uniformly Evaluating and Comparing Ranking Metrics for Spectral Fault Localization

Spectral fault localization (SFL) is one automatic fault-localization technique, which uses ranking metric to rank the risk of fault existence in each program entity after dynamically collecting the testing information. The effectiveness evaluation and comparison of ranking metrics are two important research problems. In this paper, we provide a uniformly theoretical investigation framework on longitudinally evaluating ranking metrics and horizontally comparing them for SFL techniques under any single fault scenario. We propose a generic vector table model as a novel device of thoroughly understanding various SFL techniques. By investigating rankings' mathematical formula of statements in the vector table model, the performance of different SFL techniques could be systematically analysed and compared. Under table model-driven evaluation framework, seven typical metrics as examples are explored, the existing equivalent group is extended, and the new relation of two equivalent groups is found. Our framework overcomes limitations of current empirical and theoretical approaches, and can theoretically evaluate the advantage and disadvantage of a SFL technique and compare different SFL techniques.