Morph: A Dynamically Optimized Scheduler for Large Model Training

Deep learning has become a cornerstone in artificial intelligence, prompting significant investment in large-scale model research by businesses, universities, and research institutions. As a result, the reliance on GPU clusters for deep learning research and development has intensified, underscoring the importance of efficiently allocating cluster resources. While recent advancements in deep neural network (DNN) training schedulers have improved various performance metrics, they fall short in strategies tailored for large-model training, predominantly focusing on data parallelism and overlooking the crucial role of model parallelism for large-scale models. Addressing this gap, we introduce "Morph", a novel approach that predicts throughput for model parallel tasks by evaluating all possible resource allocations. Morph leverages queueing theory to assess cluster load, optimizing resource distribution based on the cluster’s processing capabilities. Additionally, it employs a simulated annealing algorithm to refine the sequence of task queues, significantly enhancing cluster throughput. Our experiments demonstrate that Morph, integrated with existing schedulers, achieves more effective resource allocation, optimizes task sequences, and improves cluster throughput, markedly reducing average job completion time by up to 50.69% and queue waiting times by up to 71.01%.