On the Interplay Between Precision, Rank and Admissibility for Iterative Hierarchical Low-Rank Matrix Solvers

Solving large linear systems efficiently is at the heart of many problems in high-performance computing. For dense matrices with a particular structure that results in low-rank off-diagonal blocks (e.g., Green's function interaction matrices in boundary integral equations, frontal matrices in multi-frontal solvers, and covariance matrices in statistics), hierarchical matrices with shared bases can be used to reduce the cubic complexity to linear complexity. While these matrices are subdivided into a hierarchy of blocks with the majority of them being low-rank, the underlying operations during factorization are still based on dense linear algebra. As such, they should be able to exploit the performance advantages provided by mixed precision matrix engines. We develop an approach based on mixed-precision iterative solvers that is suitable for hierarchical matrices with shared bases. By reducing the approximation accuracy during the factorization we are not only able to take advantage of low-precision arithmetic, but also succeed in reducing the total number of floating-point operations (FLOPS) for a desired target accuracy. Additionally, the iterative refinement serves as a way of restoring accuracy that has been sacrificed during factorization in order to enable a higher degree of parallelism. The proposed algorithm is implemented on GPU and capable of using low-precision Tensor Cores during factorization. We investigate the trade-off between accuracy, floating-point precision, and number of iterations and provide insights on optimal choices of the individual parameters.