Sequences of Distributed Matrix-Vector Product for Very Large and Very Sparse Irregular Matrices

We study the performance behavior of the sparse matrix-vector product operation in distributed computing environments, in the case of very large non-diagonal matrices where the nonzero elements are placed irregularly across the matrix. In particular, we focus on the distributed storage of the result vector in cases where it becomes too large to be stored fully on each process, and its redistribution between the iterations of a sequence of SpMV operations. We propose two methods to this effect; one aims at minimizing the memory requirements of storing the result vector, the other optimizes the communications required for the redistribution. We perform large-scale experiments on the Fugaku supercomputer, in order to show the importance of taking into account the network topology to correctly identify and address communication bottlenecks. The results show that the most efficient proposed method has a runtime several times faster than a non-optimal one on this topology.