Analysis of Parallel Training Algorithms for Deep Neural Networks

A majority of training algorithms for deep neural networks (DNNs) use variants of stochastic gradient descent (SGD). Since a large amount of time is typically required to train DNNs, many attempts have been made to speed-up the training process by parallelizing the SGD algorithms. However, such parallelization efforts introduce approximation due to the inherent sequential nature of the SGD methods. In this paper, we revisit and analyze parallel SGD algorithms, and propose a novel pipelined SGD that is more efficient than previous algorithms.