Unified Collective Communication: A Unified Library for CPU, GPU, and DPU Collectives

Unified collective communication (UCC) is an application programming interface (API) and library implementation of collective communication operations. The goal of UCC is to provide a unified API and library serving the collective communication needs of various workloads running on a wide variety of system architectures. Particularly, we aim to unify the collective communication interfaces and semantics and provide a common implementation framework for: 1) parallel programming models, deep learning, and I/O; 2) collectives moving data in CPU main memory and device memory (GPU and data processing unit, DPU); and 3) collective operations using software point-to-point and hardware transports. In this article, we present an overview of UCC’s design, interfaces, semantics, and an implementation. We demonstrate UCC’s capabilities through evaluations with microbenchmarks representing diverse workloads and applications across various programming models, including MPI, Partitioned Global Address Space (OpenSHMEM), and PyTorch, on multiple hardware architectures (CPU, GPU, and DPU).