Two-Stage Block Orthogonalization to Improve Performance of s-step GMRES

Generalized Minimum Residual (GMRES) is a popular Krylov subspace projection method for solving a large nonsymmetric linear system of equations. It computes the approximate solution with the minimum residual norm in the generated Krylov projection subspace. On current computer architectures, the solver performance can be limited by its communication cost to generate the orthonormal basis vectors of the projection subspace. To address this potential performance bottleneck, its s-step variant orthogonalizes a block of s basis vectors at a time, providing the potential to reduce the communication cost by a factor of s. Unfortunately, for a large step size s, the solver can generate extremely ill-conditioned basis vectors, and to maintain the stability in practice, a conservatively small step size is used, which limits the performance advantages that the s-step solver can provide. To enhance the solver performance using a small step size, in this paper, we introduce a two-stage block orthogonalization scheme. Similar to the original scheme, the first stage of the proposed method operates on the block of s basis vectors at a time, but its objective is to maintain the well-conditioning of the generated basis vectors with a lower cost. The orthogonalization of the basis vectors is delayed until the second stage when enough basis vectors are generated to obtain higher performance. Our analysis shows the stability of the proposed two-stage scheme. At the same time, the proposed scheme improves the performance because while still requiring the same amount of computation as the original scheme, it reduces the overall communication requirements. Our performance results with up to 192 NVIDIA V100 GPUs on the Summit supercomputer demonstrate that when solving a 2D Laplace problem, the two-stage approach can reduce the orthogonalization time and the total time-to-solution by the respective factors of up to 2.6× and 1.6× over the original s-step GMRES, which had already obtained the respective speedups of 2.1× and 1.8× over the standard GMRES. Similar speedups were obtained for 3D problems and for matrices from the SuiteSparse Matrix Collection.