Millions of Matrix-Multiplications: GEMM Variations on Aurora

Supercomputers are typically composed of hundreds to tens of thousands of nodes. Although one may benchmark and optimize kernels on just one of those nodes, there will be hardware or manufacturing differences between each node, which can impact overall performance. Hence, it is important to understand the variation in the performance of individual components of a supercomputer.To investigate the variation, we measured the performance of GEMMs on each of 9762 nodes on the supercomputer Aurora. The performance was measured on each CPU and GPU on each node, and with multiple data types (FP64, FP32, TF32, BF16, I8). We evaluated how spread out the measurements across components are and how well they match a Gaussian distribution. We find that the variation between components is low for all GEMM types (≤ 3.0%) and most fit a Gaussian distribution well. We also find that there were some significantly poorly performing components and we discuss a method to identify and remove these outliers.We provide these results as a reference for supercomputer users, benchmarkers, and maintainers. Although the exact variations and distributions are specific to Aurora, the methods could be repeated on any large system.