SaPus: Self-Adaptive Parameter Update Strategy for DNN Training on Multi-GPU Clusters

Parameter server architecture has been identified as an efficient framework for scaling DNNs training on clusters. For large-scale deployment, communication becomes the bottleneck, and the parameter updating strategy strongly impacts the training performance and accuracy. Recent state-of-art solutions have adopted the local SGD approach, which enables workers to update their local version of models and only aggregate them to update the global parameters after finishing a number of iterations, to alleviate heavy communication pressure on the parameter server and improving the training performance. We identify three limitations of these works. First, these works do not provide an approach for determining when the worker is to update the parameter with the server under asynchronous communication strategies that can guarantee the training performance. Second, local SGD suffers from the problem of unbounded gradient delay. Previous work works well for a short delay while can not guarantee the performance with an increase of gradient delay. Third, they do not consider the system performance when determining the update interval of the local SGD, including the CPU, memory, and network, which affects the training performance extremely. We provide a self-adaptive parameter updating strategy called SaPus, which allows each worker to detect their training results through quantification of the accumulated gradient updates and determine when to update the parameter with the server adaptively and individually. Theoretical lower and upper bound of the update interval is also provided. We also propose a weighted aggregation algorithm based on a global-loss window, which is used to collect the most recent loss value of other workers to calculate a
weight
for the accumulated gradients of each worker to solve the unbounded delay problem in asynchronous local SGD. To increase the robustness of our parameter updating strategy, a performance model is built to provide a resource-aware lower bound for the update interval. Extensive experimental results generated on GPU cluster indicate that our model improves the training performance of DNNs, achieving up to
$66.67\%$
speedup as compared with state-of-art solutions. Further, results show the CPU utilization of server dropped by up to
$81.1\%$
and network bandwidth usage reduced to less than
$1~Gbps$
on an average during the training.