Toward Context-Aware Federated Learning Assessment: A Reality Check

Federated learning (FL) enabled creating models that are competitive to centralized machine learning models, without compromising user privacy. Participating FL clients train local models on their data and only share model weights. An FL server aggregates these weights into global weights that are pushed to clients for the next training round. Despite FL research growth, most of this work is conceived in experimental simulated environments that do not reflect its applicability to real-world scenarios. Also, existing open-source FL testbeds/frameworks have drawbacks that prohibit convenient deployment over a large spectrum of heterogeneous clients in realistic environments. These drawbacks include simulations, unrealistic data sets, not supporting heterogeneity, and not having realistic environment control in terms of network and client churn, for example. In this article, we introduce (RealFL) a novel, realistic, open-source, and extendable platform for FL that supports a large scale of heterogeneous clients. It enables a realistic assessment of FL solutions by controlling various environmental parameters, e.g., network, client churn, data distribution, training complexity, and client heterogeneity. Using these parameters, we assess RealFL performance through an extensive evaluation. Preliminary evaluation shows a performance gap of up to 72% in training time and 27% in accuracy between FL-simulated environments and RealFL. Moreover, extensive evaluation reveals that realistic environmental parameters could affect accuracy by up to 52.7%, training time by up to 77.5%, and communication overhead by up to 98%.