FedHCL: Hierarchical Cluster Fine-Grained Prototype Contrastive Learning for Multi-Feature Distributions and Model Heterogeneity in FL

Recently, Federated Learning (FL) has gained widespread attention for its ability to support multiple clients in collaborative learning while preserving data privacy. However, existing solutions mainly focus on the issue of the clients private data being derived from a single feature distribution. When the clients data exhibits intra-class heterogeneity, where a single class contains multiple feature distributions, using a single prototype to represent the entire class becomes inadequate. Furthermore, in cross-silo FL, feature distributions often vary significantly across clients (i.e., feature shift), posing additional challenges to effectively draw knowledge from other clients to improve the generalization capability of the local model. Moreover, FL also faces the challenge of model heterogeneity, with current methods typically assuming model homogeneity and exchanging knowledge through model parameter sharing. To address these issues, we propose a Heterogeneous Federated Learning (HtFL) approach called FedHCL, which is designed to share fine-grained prototypes—each class represented by multiple prototypes— between heterogeneous clients while maintaining the privacy of their models. To overcome the limitation of a single prototype in fully representing class knowledge, we propose a multi-finegrained prototype method to complement the single prototype. Additionally, we use Hierarchical Cluster Fine-grained Prototype Contrastive Learning (HCL) to learn trainable global finegrained prototypes on the server, replacing traditional methods, such as weighted averaging or direct averaging, to acquire global knowledge. The clients then perform local training using our carefully designed loss function to enhance model generalization. Extensive experiments on ten heterogeneous models demonstrate that our FedHCL method outperforms recent HtFL methods in terms of accuracy.