Evaluating Data-Parallel Distributed Training Strategies

Deep neural networks (DNNs) require distributed training strategies to deal with large data sizes. TensorFlow is one of the most widely used frameworks that support distributed training. Among the TensorFlow training strategies, the mirrored strategy (MS) and multi-worker mirrored strategy (MWMS) follow synchronous distributed model training; the parameter server strategy (PSS) implements asynchronous methods for training. In this work, we analyze the performance of distributed training strategies in synchronous and asynchronous settings by implementing LSTM and MLP on two industrial time-series datasets from network management systems (NMS) and Google cluster trace (GCT), and Inception-v3 model on CIFAR-10 image dataset. The strategies were compared by empirically evaluating loss, accuracy, training time, and scalability.