Development and Evaluation of High-Density Multi-GPU Sub-System Using PCI Express Expansion Hardware

In this paper, we develop and evaluate a high-density multi-GPU hardware sub-system for high performance computing and deep learning. The high-density multi-GPU hardware is implemented as an out-of-box hardware by extending PCI Express system bus via multi-Gbps class cable assemblies from a server. The high-density multi-GPU hardware extends the PCI Express in a multiple-ways and provides multiple GPUs for a server. The high-density multi-GPU hardware is developed to fit into 21 inches OCP Open Rack Vl and V2 rack and chassis having density of 12 GPUs in 4OU (Open rack Unit) height. The HPL and deep learning applications are tested to evaluate the performance of the developed high-density multi-GPU hardware. The initial test result of HPL is shown that the maximum performance of 36.22 TFLOPS (efficiency of 63.4%) with 12 NVIDIA P100 on triple sever nodes. For deep learning application, the resnet50 result of 3, 187.2 images/sec is obtained with 12 NVIDIA P100 on single sever node with synthetic data. The experimental results show that the developed multi-GPU hardware sub-system exhibits relatively good scalability in a single node especially deep learning applications than the HPL.