Performance Analysis of Communication Scheduling Schemes for Distributed Deep Learning

With the growing popularity of large-scale deep neural networks, efficient communication scheduling has become crucial in distributed deep learning systems to reduce overall training time. In multi-job distributed training scenarios, current communication scheduling methods do not effectively utilize the periodic communication patterns of deep learning training (DLT) jobs to reduce the potential link contention. When multiple tenants run concurrent jobs and compete for network resources, training performance can degrade due to increased network contention. In this paper, we focus on exploring the potential of leveraging periodic communication patterns in scheduling DLT jobs. We analyze the performance of static shift-based scheduling strategies based on the least common multiple (LCM) alignment in handling multi-job communication conflicts. Through theoretical analysis and validation via real-world experiments, we expose fundamental limitations of shift-based scheduling strategies, which fail to improve training throughput in about 73 % of multi-job scenarios. Our research work provides guidance for future research on understanding traffic patterns of DLT jobs and lays the groundwork for communication scheduling optimization in multi-tenant clusters.