Optimization of Collective Communication for Heterogeneous HPC Platforms

This chapter overviews the existing research in collective communication area. Observes both message passing interface (MPI) collectives as well as alternatives from the distributed computing domain. It reviews the main generic optimization techniques for collective communication. Most optimizations of collective operations on homogeneous clusters focus on finding an efficient algorithm on top of point‐to‐point primitives. With increasingly heterogeneous networks, empirical approaches to optimizing communication become unfeasible because of the exponential growth of the already huge test space. Therefore, it needs some sort of network model. Such a network model based on topology or performance. Topology‐aware collectives are a relatively straightforward and popular approach to optimization. The minimal spanning trees based on the per‐link Hockney model provide efficient broadcast for small messages. For very large messages, receiver‐initiated multicasts are gaining popularity in the high‐performance computing (HPC) domain. The adaptive nature of these algorithms makes them suitable even for very complex networks.