Wyner-Ziv compression is (almost) optimal for distributed optimization

Consider distributed optimization of smooth convex functions over ℝ
^d
where K independent clients can provide estimates of the gradient. Assume that all the gradient estimates are within Euclidean distance σ of the true gradient and that each oracle’s output must be compressed to r bits. For this problem, in the centralized setting with one client, the optimal convergence rate using T iterations is known to be roughly $\sqrt {{\sigma ^2}/T} $. We show that in the distributed setting the optimal convergence rate for large K is roughly $\sqrt {{\sigma ^2}/T} \cdot \sqrt {d/Kr} $. Our main contribution is an algorithm which attains this rate by exploiting the fact that the gradient estimates are close to each other. Specifically, our gradient compression scheme first uses half of the parties to form side information and then uses a Wyner-Ziv compression scheme to compress the remaining half of the gradient estimates.