Paradoxes in Numerical Comparison of Optimization Algorithms

Numerical comparison is often key to verifying the performance of optimization algorithms, especially, global optimization algorithms. However, studies have so far neglected issues concerning comparison strategies necessary to rank optimization algorithms properly. To fill this gap for the first time, we combine voting theory and numerical comparison research areas, which have been disjoint so far, and thus extend the results of the former to the latter for optimization algorithms. In particular, we investigate compatibility issues arising from comparing two and more than two algorithms, termed “C2” and “C2+” in this article, respectively. Through defining and modeling “C2” and “C2+” mathematically, it is uncovered and illustrated that numerical comparison can be incompatible. Further, two possible paradoxes, namely, “cycle ranking” and “survival of the nonfittest,” are discovered and analyzed rigorously. The occurrence probabilities of these two paradoxes are also calculated under the no-free-lunch assumption, which shows the first justifiable use of the impartial culture assumption from voting theory, providing a point of reference to the frequency of the paradoxes occurring. It is also shown that significant influence on these probabilities comes from the number of algorithms and the number of optimization problems studied in the comparison. Further, various limiting probabilities when the number of optimization problems goes to infinity are also derived and characterized. The results would help guide benchmarking and developing optimization and machine learning algorithms.