Harnessing the Power of Local Supervision in Federated Learning

Federated learning is widely accepted as a privacy-preserving paradigm for training a shared global model across multiple client devices in a collaborative fashion. However, in practice, the significantly limited computational power on client devices has been a major barrier when we wish to train large models with potentially hundreds of millions of parameters. In this paper, we propose a new architecture, referred to as
Infocomm
, that incorporates locally supervised learning in federated learning. With locally supervised learning, the disadvantages of split learning can be avoided by using a more flexible way to offload training from resource constrained clients to a more capable server.
Infocomm
enables parallel training of different modules of the neural network in both the server and clients in a gradient-isolated fashion. The efficacy in reducing both training time and communication time is supported by our theoretical analysis and empirical results. In the scenario involving larger models and fewer available local data,
Infocomm
has been observed to reduce the elapsed time per round by over 37% without sacrificing accuracy compared to both conventional federated learning or directly combining federated learning and split learning, which showcases the advantages of
Infocomm
under power-constrained IoT scenarios.