HPL and DGEMM Performance Variability on the Xeon Platinum 8160 Processor

Initial testing of a cluster equipped with Intel Xeon Platinum 8160 processors showed occasional slow single-node HPL benchmark performance - approximately 0.4% of single- node results were more than 10% slower than expected. We describe a systematic series of experiments using simplified benchmarks and hardware performance counters, showing that the increased run times were associated with increased DRAM traffic, that this was caused by increased L2 cache miss rates, and that these were caused by snoop filter evictions. These evictions resulted from associativity conflicts in the snoop filters, and were traced to pathological interactions of data physical addresses with the hash function that distributes addresses across the coherence agents on the processor. For the HPL benchmark, switching from 2MiB hugepages to 1GiB hugepages eliminated the conflict and the associated slowdowns. The snoop filter conflict was later reproduced using a simple array summation kernel, suggesting that other applications could be impacted.