A Study in Prioritization for Higher Strength Combinatorial Testing

Recent studies have shown that combinatorial interaction testing (CIT) is an effective fault detection technique and that early fault detection can be improved by ordering test suites by interaction based prioritization approaches. Despite research that has shown that higher strength CIT improves fault detection, there have been fewer studies that aim to understand the impact of prioritization based on higher strength criteria. In this paper, we aim to understand how interaction based prioritization techniques perform, in terms of early fault detection when we prioritize based on 3-way interactions. We generalize prior work on prioritizing using 2-way interactions to t-way prioritization, and empirically evaluate this on three open source subjects, across multiple versions of each. We examine techniques that prioritize both existing CIT suites as well as generate new ones in prioritized order. We find that early fault detection can be improved when prioritizing 3-way CIT test suites by interactions that cover more code, and to a lesser degree when generating tests in prioritized order. Our techniques that work only from the specification, appear to work best with 2-way generation.