Multi-Message Gradient Coding for Utilizing Non-Persistent Stragglers

Due to large variability in modern distributed systems, large scale machine learning suffers from slow workers, i.e. stragglers, that can significantly decrease the speed of computation. Redundancy based approaches have been proposed to solve this issue but many of these schemes under-utilize the work performed by stragglers due to allowing only one communication message per worker. In this paper, we propose a Multi-Message Gradient Coding scheme to exploit the work done by stragglers. Our approach provides a trade-off between time to completion and communication load without any data encoding. We analyze our construction by providing a characterization of expected time to completion and communication load. Additionally, we perform simulations to demonstrate the flexibility of our approach.Review Area: D.8 Distributed Computation and Storage