Basil: A Fast and Byzantine-Resilient Approach for Decentralized Training

Decentralized (i.e., serverless) training across edge nodes can suffer substantially from potential Byzantine nodes that can degrade the training performance. However, detection and mitigation of Byzantine behaviors in a decentralized learning setting is a daunting task, especially when the data distribution at the users is heterogeneous. As our main contribution, we propose
Basil
, a fast and computationally efficient Byzantine-robust algorithm for decentralized training systems, which leverages a novel sequential, memory-assisted and performance-based criteria for training over a logical ring while filtering the Byzantine users. In the IID dataset setting, we provide the theoretical convergence guarantees of
Basil
, demonstrating its linear convergence rate. Furthermore, for the IID setting, we experimentally demonstrate that
Basil
is robust to various Byzantine attacks, including the strong Hidden attack, while providing up to absolute ~16% higher test accuracy over the state-of-the-art Byzantine-resilient decentralized learning approach. Additionally, we generalize
Basil
to the non-IID setting by proposing Anonymous Cyclic Data Sharing (ACDS), a technique that allows each node to anonymously share a random fraction of its local non-sensitive dataset (e.g., landmarks images) with all other nodes. Finally, to reduce the overall latency of
Basil
resulting from its sequential implementation over the logical ring, we propose
Basil+
that enables Byzantine-robust parallel training across groups of logical rings, and at the same time, it retains the performance gains of
Basil
due to sequential training within each group. Furthermore, we experimentally demonstrate the scalability gains of
Basil+
through different sets of experiments.