Towards Addressing Heterogeneity Of Data In Federated Learning

Federated Learning (FL) aims to train a centralized model that learns from different clients via communications without sharing the clients' local data. However, one major challenge of federated learning exists in most task scenarios. Data heterogeneity refers to the difference in the local data distribution between various clients. In this paper, we introduce the heterogeneity of data in the federated learning problem, where the local data by clients is likely to come from different distributions, such as agricultural machines responsible for different regions, which may face different environmental distributions. We first analyze the influence of different heterogeneous data on the model in local training. Interestingly, we find that profound heterogeneity of data will lead to the collapse of the distribution of local models in the output feature. To solve this problem, we propose a covariance constraint method based on output feature distribution(FedOFCC), an effective method to alleviate the influence of data heterogeneity on the model representation in federated learning. The experimental results on several benchmark data sets demonstrate that our method is superior to the baseline.