Neighbor Selecting Cooperation Strategies for Distributed Learning

In distributed learning, a network of agents co-operate for solving a common task, like training a particular neural network. The devices usually adopt an iterative procedure with two steps, namely, they, first, perform local optimization, using, e.g., stochastic gradient descent, and, then, they exchange information among them in order to achieve consensus on the final solution. In current literature, the proposed distributed algorithms achieve consensus using averaging rules over the received information at each agent. Here, the paper departs from this paradigm and focuses on “single agent” cooperation strategies in which each agent selects a particular neighbor at each iteration and uses only that information during the local optimization step. Three selection rules are designed and it is shown that they inherit the convergence properties of commonly used averaging rules. Moreover, their effectiveness is demonstrated experimentally in classification tasks over other algorithms using well-known datasets for a wide range of scenarios, capturing factors like non-IID datasets, network size, and AI model size.