SPinS-FL: Communication-Efficient Federated Subnetwork Learning

Federated learning (FL) is a distributed machine learning method in which edge devices collaboratively train a unified model without disclosing their private training data to others. Unlike data centers, edge devices often stand in low-bandwidth traffic environments, which can be a bottleneck in the training process. To tackle this problem, we first consider a naive FL based on the Edge-Popup algorithm. The Edge-Popup-based neural network constructs a subnetwork by assigning a score to each randomly initialized weight and selecting weights based on the score without updating the weights. Forward propagation, based on the subnetwork consisting of only the live weights with high scores, is performed, and then backpropagation is performed to update scores and search for a subnetwork that achieves high accuracy. This algorithm has two unique properties: first, the score itself is unnecessary for calculating the score's gradient, as long as there is a supermask, a bitmask indicating which weights are participating in the subnetwork then; and second, the intensity of the turnover in the score rankings varies according to the scores' rank range. Exploiting these properties, we propose SPinS-FL, a novel FL method based on the Edge-Popup algorithm. Instead of communicating all the scores with the server, SPinS-FL only communicates some supermasks and a partial set of scores of the weights around the boundary between selected and unselected in the subnetwork; this dramatically improves communication efficiency. Numerical experiments using the CIFAR-10 and CIFAR-100 datasets show that SPinS-FL succeeds in reducing the traffic volume by up to 79.8% compared with communicating all the scores. Moreover, SPinS-FL achieves comparable accuracy as the federated averaging method but with up to 69.9% less traffic cost.