Efficient Adapting for Vision-language Foundation Model in Edge Computing Based on Personalized and Multi-Granularity Federated Learning

Foundation model (FM) has shown great potential in various downstream tasks, such as CLIP, a vision-language pre-trained model that effectively aligns semantics in text and image spaces. Federated learning (FL), as a distributed training paradigm, can provide adequate data for adapting FM while overcoming challenges such as communication pressure and user privacy leakage. However, in edge computing with a large scale of edge devices, vanilla FL faces challenges, including limited computation resources, inadequate bandwidth, non-IID and multi-granularity data caused by the diversity of devices and users. To feasibly and efficiently collaborate with the large and diverse edge devices in adapting FM, we propose a novel personalized and multi-granularity FL framework (PMG-FL) that provides a personalized lightweight prompt for each edge device while considering interactions among same-granularity edge devices and cross-granularity edge devices. In particular, we introduce prompt training to adapt FM locally, which can mitigate computation and communication pressure with few learnable prompt parameters. Based on the prompt parameters, we design a distance-based prompt aggregation mechanism to capture similarities among same-granularity edge devices and aggregate the personalized prompt for each edge device, addressing the challenges posed by non-IID data. Furthermore, we design a cross-granularity guidance mechanism that leverages the correlation of semantic knowledge among edge devices with multigranularity data. Extensive experimental results demonstrate the superiority of PMG-FL over the alternative approaches, with robust performance on both IID and non- IID data.