Multi-Objective Configuration Sampling for Performance Ranking in Configurable Systems

The problem of performance ranking in configurable systems is to find the optimal (near-optimal) configurations with the best performance. This problem is challenging due to the large search space of potential configurations and the cost of manually examining configurations. Existing methods, such as the rank-based method, use a progressive strategy to sample configurations to reduce the cost of examining configurations. This sampling strategy is guided by frequent and random trials and may fail in balancing the number of samples and the ranking difference (i.e., the minimum of actual ranks in the predicted ranking). In this paper, we proposed a sampling method, namely MoConfig, which uses multi-objective optimization to minimize the number of samples and the ranking difference. Each solution in MoConfig is a sampling set of configurations and can be directly used as the input of existing methods of performance ranking. We conducted experiments on 20 datasets from real-world configurable systems. Experimental results demonstrate that MoConfig can sample fewer configurations and rank better than the existing rank-based method. We also compared the results by four algorithms of multi-objective optimization and found that NSGA-II performs well. Our proposed method can be used to improve the ranking difference and reduce the number of samples in building predictive models of performance ranking.