Benchmark Set Reduction for Cheap Empirical Algorithmic Studies

The present paper introduces a benchmark set reduction strategy that can degrade the experimental evaluation cost for the algorithmic studies. Algorithm design is an iterative development process within in a test-revise loop. Starting a devised algorithm's initial version, it needs to be tested for revealing its advantages and drawbacks. Its shortcomings lead the designers to modify the algorithm. Then, the modified algorithm is again tested. This two-step pipeline is repeated until the algorithm meets the expectations such as delivering the state-of-the-art results on a set of benchmarks for a specific problem. That recurring testing step can be a burden for whom with limited computational resources. The mentioned computational cost can mainly occur due either high per instance runtime cost or having a large benchmark set. This study focuses on the cases when a target algorithm needs to be assessed across large benchmark sets. The idea is to automatically extract problem instance representation through an instance-algorithm performance data. The derived representation in the form of latent features is utilized to determine a small yet a representative subset of a given large instance set. The proposed strategy is investigated on the Traveling Thief Problem of 9720 instances. The corresponding performance data is collected by the help of 21 TTP algorithms. The resulting computational analysis showed that the proposed method is capable of substantially minimizing the benchmark instance set size.