Distributed Heuristic Adaptive Neural Networks With Variance Reduction in Switching Graphs

This article proposes a distributed adaptive training method for neural networks in switching communication graphs to deal with the problems concerned with massive data or privacy-related data. First, the stochastic variance reduced gradient (SVRG) is used for the training of neural networks. Then, the authors propose a heuristic adaptive consensus algorithm for distributed training, which adaptively adjusts the weighted connectivity matrix based on the performance of each agent over the communication graph. Furthermore, it is proved that the proposed distributed heuristic adaptive neural networks ensure the convergence of all the agents to the optimum with a single communication among connected neighbors after every training step, which is also suitable for switching graphs. This theorem is verified by the simulation, which gives the results that fewer iterations are required for all agents to reach the optimum using the proposed heuristic adaptive consensus algorithm, and the SVRG can greatly decrease the fluctuations caused by the stochastic gradient and improve its performance with only a little extra computational cost.