Going beyond Mean and Median Programs Performances

In the area of code performance optimisation and tuning, we are faced on the difficult problem of selecting the "best" code version based on empirical experiments and statistical analysis. With the massive introduction of general purpose multicore processors, programs performances become more and more instable, especially parallel programs. Usual statistical methods for computing performance speedups and comparing between programs are based on testing mean or median values. In this article, we explain why these metrics may be inadequate for making relevent decisions, and we propose new performance metrics based on parametric statistics using gaussian mixture models. Our new statistical methods are more accurate for decision making, they are formally defined, computed, implemented and distributed as free software in [1].