A Preliminary Scalability Analysis of SPEC CPU2017 Benchmarks

The SPEC CPU benchmark suites of compute-intensive benchmarks are widely used for evaluating performance of modern computer systems. Following a shift from single-core to multi-core processors, the latest suites, SPEC CPU2017, include benchmarks that are multi-threaded. This paper explores the scalability of CPU2017 benchmarks by investigating their performance in modern computer systems as a function of the number of threads for speed benchmarks and the number of copies for rate benchmarks. Based on the experimental study conducted on a pair of in-house machines, we categorize benchmarks into groups that `scale-well', `scale-moderately', and `scale-poorly'. In addition, we analyze the publicly available SPEC CPU2017 results to explore scalability trends of individual benchmarks on a wider variety of computer systems. The main findings of the study are as follows: (a) A number of speed multi-threaded benchmarks that use OpenMP saturate after a certain number of threads. The paper provides the maximum speedups and thread count needed to reach saturation. (b) Memory intensive rate benchmarks scale poorly because their demands for shared resources exceed the capacity of modern computer systems. The study aims to inform performance analysts of the limitations of the speed benchmarks in extracting system parallelism and to provide scalability expectations for the rate benchmarks on modern computer systems.