Communication-Efficient Distributed SGD Using Random Access for Over-the-Air Computation

In this paper, we study communication-efficient distributed stochastic gradient descent (SGD) with data sets of users distributed over a certain area and communicating through wireless channels. Since the time for one iteration in the proposed approach is independent of the number of users, it is well-suited to scalable distributed SGD. Furthermore, since the proposed approach is based on preamble-based random access, which is widely adopted for machine-type communication (MTC), it can be easily employed for training models with a large number of devices where MTC is used for their connectivity. For fading channel, we show that noncoherent combining can be used. As a result, no channel state information (CSI) estimation is required. For over-the-air computation, the proposed approach takes advantage of random access, where the access probability is to encode the norm of local gradient vector (without using any additional bits) and the preamble is to encode the quantized normalized gradient vector. From analysis and simulation results, we can confirm that the proposed approach is not only scalable, but also provides improved performance as the number of devices increases. From analysis and simulation results, we can confirm that the proposed approach is not only scalable, but also provides improved performance as the number of devices increases.