On the Fault-Detection Capabilities of Adaptive Random Test Case Prioritization: Case Studies with Large Test Suites

An adaptive random (AR) testing strategy has recently been developed and examined by a growing body of research. More recently, this strategy has been applied to prioritizing regression test cases based on code coverage using the concepts of Jaccard Distance (JD) and Coverage Manhattan Distance (CMD). Code coverage, however, does not consider frequency, furthermore, comparison between JD and CMD has not yet been made. This research fills the gap by first investigating the fault-detection capabilities of using frequency information for AR test case prioritization, and then comparing JD and CMD. Experimental results show that "coverage" was more useful than "frequency" although the latter can sometimes complement the former, and that CMD was superior to JD. It is also found that, for certain faults, the conventional "additional" algorithm (widely accepted as one of the best algorithms for test case prioritization) could perform much worse than random testing on large test suites.