Statistical Models for the Analysis of Optimization Algorithms With Benchmark Functions

Frequentist statistical methods, such as hypothesis testing, are standard practices in studies that provide benchmark comparisons. Unfortunately, these methods have often been misused, e.g., without testing for their statistical test assumptions or without controlling for familywise errors in multiple group comparisons, among several other problems. Bayesian data analysis (BDA) addresses many of the previously mentioned shortcomings but its use is not widely spread in the analysis of empirical data in the evolutionary computing community. This article provides three main contributions. First, we motivate the need for utilizing BDA and provide an overview of this topic. Second, we discuss the practical aspects of BDA to ensure that our models are valid and the results are transparent. Finally, we provide five statistical models that can be used to answer multiple research questions. The online Appendix provides a step-by-step guide on how to perform the analysis of the models discussed in this article, including the code for the statistical models, the data transformations, and the discussed tables and figures.