Prototype-Based Clustered Federated Learning for Semantic Segmentation of Aerial Images

Despite its impressive performance on semantic segmentation of remote sensing imagery, deep learning requires a large amount of labeled data for model training, which is both laborious and timeconsuming for an individual institution. To cope with this obstacle, federated Learning (FL) has been proposed to enable multiple institutions to train a global model collaboratively without violating privacy rules. However, the performance of FL is poor in the presence of heterogeneous training data, i.e. the data is not independently and identically distributed (non-i.i.d) among participating clients, especially for remote sensing images with high spatial and spectral heterogeneity. In this paper, we propose an FL algorithm combined with prototype-based hierarchical clustering (Fed-PHC). Instead of updating a single global model to capture the shared knowledge of all clients, we utilize a mixture of multiple global models to handle the heterogeneity between various clients using hierarchical clustering (HC) based on the prototypical representations of clients' datasets. As a result, FedPHC can reduce the domain discrepancy within each group and obtain more representative models for heteroge-neous datasets. Extensive experiments on the Inria Aerial Image Dataset confirm the effectiveness of FedPHC.