On Maintaining Linear Convergence of Distributed Learning and Optimization under Limited Communication

In parallel and distributed machine learning multiple nodes or processors coordinate to solve large problems. To do this, the nodes need to compress important algorithm information to bits so they can communicate. The goal of this paper is to explore how we can maintain the convergence of distributed algorithms under such compression. In particular, we consider a general class of linearly convergent parallel/distributed algorithms and illustrate how we can design quantizers compressing the communicated information to a few bits while still preserving the linear convergence. We illustrate our results on distributed learning algorithms and numerically implement our results using real-world data.