Leveraging Memory Expansion to Accelerate Large-Scale DL Training

Modern Deep Learning (DL) models require massive clusters of specialized, high-end nodes to train. Designing such clusters to maximize both performance and utilization is a challenging task requiring careful balance of compute, memory, and network resources. Moreover, a plethora of each model's tuning knobs drastically affect a cluster's performance, with optimal values often depending on the underlying hardware characteristics, thus necessitating a complex cluster-workload co-design process. To facilitate design space exploration, we develop Come T,a holistic cluster design methodology and workflow to jointly study the impact of parallelization strategies and key cluster resource provisioning on the performance of distributed DL training. In this short paper, we use Comet to focus on studying the impact of each compute node's memory system on overall cluster performance. We show that per-node memory expansion using technologies like CXL is a promising technique to boost a cluster's DL training performance.