A Smoothed Bregman Proximal Gradient Algorithm for Decentralized Nonconvex Optimization

Decentralized computation has received considerable research interest lately, due to its wide applications in information processing systems. However, one key requirement to establish convergence for almost all decentralized algorithms, for convex and non-convex problems alike, is that the loss function has Lipschitz-continuous gradient (LipGrad). This is a strong assumption, which does not hold for many practical problems, such as matrix/tensor factorization, neural network training, etc. On the contrary, in the centralized setting, one can utilize techniques such as the Bregman proximal gradient (BPG) method to deal with the lack of LipGrad. This work fills the gap between centralized and decentralized cases by developing a novel smoothed decentralized BPG algorithm to deal with a class of nonconvex decentralized problem, where the local problems do not have LipGrad objective functions. By leveraging the recent notion of relative smoothness and primal-dual error bounds, we show that the proposed algorithm achieves a certain ε-stationary solution by using $\mathcal{O}\left( {{\varepsilon ^{ - 2}}} \right)$ iterations, matching the rate of the centralized Bregman proximal gradient method. To our knowledge, this is the first decentralized algorithm that matches the centralized convergence rate bounds under the class of considered problems. Our numerical results on the decentralized quadratic regression example demonstrate the effectiveness of proposed algorithm.