Coverage Metrics for T-Wise Feature Interactions

Software is typically configurable by means of compile-time or runtime variability. As testing every valid configuration is infeasible, T-Wise sampling has been proposed to systematically derive a relevant subset of the configurations for testing to cover interactions among t features. Practitioners started to apply T-Wise sampling algorithms, but can often only test samples partially due to restricted resources and compare those partial samples based on their T-Wise coverage. However, there is no consensus in the literature on how to compute the T-Wise coverage in the literature. We propose the first systematic framework to define coverage metrics for T-Wise feature interactions. These metrics differ in the features and feature interactions being considered. We found evidence for at least six different metrics in the literature. In an empirical evaluation, we show that for a partial sample the coverage differs up to 21 % and for some metrics only half of the feature interactions need to be covered. As a long-term impact, our work may help to improve the efficiency and effectiveness of both, T-Wise sampling and coverage computations.