Summarizing performance is no mean feat [computer performance analysis]

For decades, computer benchmarkers have fought a war of means, arguing over proper uses of arithmetic, harmonic, and geometric means, starting in the mid-1980s. One would think this basic issue of computer performance analysis would have been long resolved, but contradictions are still present in some excellent and widely-used textbooks. This paper offers a framework that resolves these issues and includes both workload analyses and relative performance analyses (such as SPEC or Livermore Loops), emphasizing differences between algebraic and statistical approaches. In some cases, the lognormal distribution is found to be quite useful, especially with appropriate forms of standard deviation and confidence interval used to augment the usual geometric mean. Results can be used to indicate the relative importance of careful workload analysis.