GPU Acceleration of the HemeLB Code for Lattice Boltzmann Simulations in Sparse Complex Geometries

We present an implementation and scaling analysis of a GPU-accelerated kernel for HemeLB, a high-performance Lattice Boltzmann code for sparse complex geometries. We describe the structure of the GPU implementation and we study the scalability of HemeLB on a GPU cluster under normal operating conditions and with real-world application cases. We investigate the effect of CUDA block size and GPU over-subscription on the single-GPU performance, and we present a strong-scaling analysis of multi-GPU parallel simulations using two different hardware models (P100 and V100) and a variety of large cerebral aneurysm geometries. We find that HemeLB-GPU achieves single-GPU speedups of 50× (P100) and 100× (V100) compared to a single CPU core, with good scalability up to 32 GPUs. We also discuss strategies to improve both the kernel performance as well as the scalability of HemeLB-GPU to a larger number of GPUs. The GPU implementation supports the LBGK collision kernel, boundary conditions for walls and inlets/outlets, and several lattice types (D3Q15, D3Q19, D3Q27), and it integrates seamlessly with the existing infrastructure in HemeLB for graph partitioning and parallelization via MPI. It is expected that the GPU implementation will enable users of the HemeLB code to make better utilization of heterogeneous high-performance computing systems for large-scale lattice Boltzmann simulations.