Compressed Cannon's Algorithm

Parallel matrix-matrix multiplication (GEMM) is a linear-algebra operation underpinning many applications in high-performance and scientific computing. With strong scaling, GEMM's communication latency among processes, nodes, and accelerators can be a leading bottleneck in performance. Data-compression is a common solution for reducing transport costs, but is often not applied for this purpose in HPC, where low-latency & high-throughput networks are difficult for com-pressors to match. However, some parallel GEMM algorithms - such as Cannon's method - exhibit a communication pattern where data is rarely altered as it travels between processes, providing an opportunity for a simple compression strategy: compress submatrices once at the outset, and then circulate the compressed version. This minimizes compression, which is typically significantly slower than decompression, at a price of additional memory utilization. In this work, we have measured and modeled scaling studies of this strategy, using six existing and widely-used lossless compressors (NDZip, ZStandard, Zip, BloscLZ, ZFP, & LZ4). We find most of these compressors can improve communication latency, with speedups of up to 1.2 times observed. Additionally, we evaluate and discuss the accuracy of our model against our measured results.