SPARQ-SGD: Event-Triggered and Compressed Communication in Decentralized Optimization

In this article, we propose and analyze SParsified Action Regulated Quantized–Stochastic Gradient Descent (SPARQ-SGD), a communication-efficient algorithm for decentralized training of large-scale machine learning models over a graph with
$n$
nodes, where communication efficiency is achieved using
compressed
exchange of local model parameters among neighboring nodes, which is
triggered
only when an
event
(a locally computable condition) is satisfied. Specifically, in SPARQ-SGD, each node takes a fixed number of local gradient steps and then checks if the model parameters have significantly changed compared to its last update; only when the change is beyond a certain threshold (specified by a design criterion), it compresses its local model parameters using both quantization and sparsification and communicates them to its neighbors. We prove that SPARQ-SGD converges as
$O(\frac{1}{nT})$
and
$O(\frac{1}{\sqrt{nT}})$
in the strongly convex and nonconvex settings, respectively, matching the convergence rates of plain decentralized SGD. This demonstrates that we get communication efficiency achieved by aggressive compression, local iterations, and event-triggered communication essentially for
free
. We evaluate SPARQ-SGD over real datasets to demonstrate significant amount of savings in communication over the state-of-the-art while achieving similar performance.