Machine Learning over Networks: Co-design of Distributed Optimization and Communications

This paper considers a general class of iterative algorithms performing a distributed training task over a network where the nodes have background traffic and communicate through a shared wireless channel. Focusing on the carrier-sense multiple access with collision avoidance (CSMA/CA) as the main communication protocol, we investigate the mini-batch size and convergence of the training algorithm as a function of the communication protocol and network settings. We show that, given a total latency budget to run the algorithm, the training performance becomes worse as either the background traffic or the dimension of the training problem increases. We then propose a lightweight algorithm to regulate the network congestion at every node, based on local queue size with no explicit signaling with other nodes, and demonstrate the performance improvement due to this algorithm. We conclude that a co-design of distributed optimization algorithms and communication protocols is essential for the success of machine learning over wireless networks and edge computing.