Handling Non-IID Data in Federated Learning: An Experimental Evaluation Towards Unified Metrics

Recent research has demonstrated that Non-Identically Distributed (Non-IID) data can negatively impact the performance of global models constructed in federated learning. To address this concern, multiple approaches have been developed. Nonetheless, previous research lacks a cohesive overview and fails to uniformly assess these strategies, resulting in challenges when comparing and choosing relevant options for real-world scenarios. This study presents a structured survey of cutting-edge techniques for handling the Non-IID data, accompanied by proposing a metric to develop a standardized approach for assessing data skew and its harmony with the appropriate approach. The findings affirm the metric's suitability as a heuristic for assessing data skew in distributed datasets without having insight into client data, serving both scientific and practical purposes and thus supporting the selection of handling strategies. This preliminary research establishes the foundation for discussing standardizing methodologies for evaluating data heterogeneity in federated learning.