Graph-Attention-Based Decentralized Edge Learning for Non-IID Data

In conventional decentralized learning, multiple agents collaboratively build a common machine learning model. However, due to the underlying non-i.i.d. data distribution among agents, the unified learning model becomes inefficient for each agent to process its locally accessible data. To address this problem, we propose a graph-attention-based personalized training algorithm (GATTA) for decentralized deep learning. The GATTA enables each agent to train its local personalized model while exploiting its correlation with neighboring nodes and utilizing their useful information for aggregation. In particular, the personalized model in each agent is composed of a global part and a node-specific part. By treating each agent as one node in a graph and the node-specific parameters as its features, the benefits of the graph attention mechanism can be inherited. Instead of aggregation simply based on averaging, GATTA learns the specific weights for different neighboring nodes without requiring any prior knowledge about the graph structure or the neighboring nodes' data distributions. Numerical results validate the effectiveness of the proposed algorithm under different datasets as well as various network topologies.