Accelerated Primal-Dual Algorithm for Distributed Non-convex Optimization

This paper investigates accelerating the convergence of distributed optimization algorithms on non-convex problems. We propose a distributed stochastic gradient primal-dual with fixed parameters and “powerball” (DSGPA-F-PH) method to accelerate. We show that the proposed algorithm achieves the linear speedup convergence rate
$\mathcal{O}(1/\sqrt{nT})$
for general smooth (possi-bly non-convex) cost functions. We demonstrate the efficiency of the algorithm through numerical experiments by training two-layer fully connected neural networks and convolutional neural networks on the MNIST dataset to compare with state-of-the-art distributed stochastic gradient descent (SGD) algorithms and centralized SGD algorithms.