Communication-Efficient Approximate Gradient Coding Using Structured Matrices

Large scale distributed learning aims at minimizing a loss function $L$ that depends on a training dataset with respect to a $d$-length parameter vector. The distributed cluster typically consists of a parameter server (PS) and multiple workers. Gradient coding is a technique that makes the learning process resilient to straggling workers. It introduces redundancy within the assignment of data points to the workers and uses coding theoretic ideas so that the PS can recover $\nabla L$ exactly or approximately, even in the presence of stragglers. Communication-efficient gradient coding allows the workers to communicate vectors of length smaller than $d$ to the PS, thus reducing the communication time. While there have been schemes that address the exact recovery of $\nabla L$ within communication-efficient gradient coding, to our best knowledge the approximate variant has not been considered in a systematic manner. In this work, we present constructions of communication-efficient approximate gradient coding schemes. Our schemes use structured matrices that arise from bipartite graphs, combinatorial designs and strongly regular graphs, along with randomization, and algebraic constraints. Moreover, we derive a corresponding lower bound on the approximation error of any scheme. Numerical experiments demonstrate that our schemes have low approximation error. Furthermore, in several cases, we are able to provide analytical upper bounds on the approximation error of our schemes.