Bridging the Chasm Between Ideal and Realistic Federated Learning: A Measurements Study

Federated Learning is being hailed as a privacy-preserving machine learning alternative, by allowing models to be distributively trained on source devices owning their data. Most FL solutions, and their assessments, however, assume superior environmental reliability, despite the more realistic variances in environmental factors such as device and network capacity, data distribution, and device churn. As such, we argue in this paper, that there is a growing chasm between current FL assessment setups and the evolving FL assessment needs. Motivated by this chasm, we conduct, to the best of our knowledge, the first empirical measurement study of FL performance given realistic environmental factors. Our study quantifies the impact of these environmental factors on FL performance in terms of training time, accuracy, and communication overhead. Our findings have broad implications for the future development of FL including client admission control and scheduling optimizations.