Communication-efficient Federated Learning Framework with Parameter-Ordered Dropout

Large-scale models, also referred to as pretrained models, have attracted significant attention due to their outstanding performance and robust generalization. However, the high demands for data quality, stringent data privacy and security requirements, and limited computational and communication resources have imposed restrictions on the further development of large-scale models. Federated Learning (FL) is a popular machine learning framework that can effectively address this issue. However, when collaboratively training large-scale models using FL, local training and the transmission of large-scale parameters impose significant computational and communication burdens on mobile devices. This paper proposes a light-weight and high-efficiency federated learning framework (FedLH) for large-scale models. This framework divides large-scale models into semantic block-based submodels, allowing clients to transmit these submodels to the server for heterogeneous aggregation. This approach enables both communication and computational efficiency. With the proposed framework, each device can learn personalized, structured sparse models that can efficiently run on terminal devices. Experimental results demonstrate that FedLH outperforms other baseline algorithms by significantly reducing the number of training parameters and transmitted data. It also exhibits strong generalization and scalability.