Holistic Evaluation Metrics for Federated Learning

Federated Learning (FL) is a promising technology that enables multiple participants collaboratively train a joint model without sharing their local data. Owing the privacy protection nature, FL has attracted interests from industry, leading to its deployment across diverse domains such as smartphones, institutions, and Internet of Things (IoTs). Despite the development of various FL algorithms aimed at enhancing FL performance from multiple perspectives, their evaluation typically hinges on a single metric like accuracy, failing to account for the unique demands of different use cases. Thus, how to comprehensively evaluate an FL algorithm and determine the most suitable candidate for a designated use case remains an open question. To address this research gap, we introduce the Holistic Evaluation Metrics (HEM) for federated learning. In this work, we consider the application scenarios of IoT, Smartphones, and Institutions as the principal FL use cases. Subsequently, we determine the components of the evaluation metric and the corresponding importance vector for each use case. The HEM index is then generated by integrating these metric components with their importance vectors. We evaluate various FL algorithms in different use cases through HEM, our experimental results demonstrate that HEM can effectively evaluate and select the appropriate FL algorithms in different use cases.