Substate Profiling for Enhanced Fault Detection and Localization: An Empirical Study

Researchers have used execution profiles to enable coverage-based techniques in areas such as defect detection and fault localization. Typical profile elements include functions, statements, and branches, which are structural in nature. Such elements might not always discriminate failing runs from passing runs, which renders them ineffective in some cases. This motivated us to investigate alternative profiles, namely, substate profiles that aim at approximating the state of a program (as opposed to its execution path). Substate profiling is a recently presented form of state profiling that is practical, fine-grained, and generic enough to be applicable to various profile-based analyses. This paper presents an empirical study demonstrating how complementing structural profiles with substate profiles would benefit Test Suite Reduction (TSR), Test Case Prioritization (TCP), and Spectrum-based Fault Localization (SBFL). Using the Defects4J benchmark, we contrasted the effectiveness of TSR, TCP, and SBFL when using the structural profiles only to when using the concatenation of the structural and substate profiles. Leveraging substate profiling enhanced the effectiveness of all three techniques. For example: 1) For TSR, 86 more versions exhibited 100% defect detection rate. 2) For TCP, 22 more versions had one of their failing tests ranked among the top 20%. 3) For SBFL,substate profiling localized 14 faults that structural profiling failed to localize. Furthermore, our study showed that the improvement due to substate profiling was noticeably more significant in the presence of coincidentally correct tests than in their absence. This positions substate profiling as a promising basis for mitigating the negative effect of coincidental correctness.