CP2GFed: Cross-granular and Personalized Prompt-based Green Federated Tuning for Giant Models

Giant models have transformed vision-language tasks by mastering consistent representations across text and images, highlighting the critical role of deploying such models in the expanding domain of edge scenarios, such as monitoring and segmentation. However, the deployment is challenged by device heterogeneity, limited computational resources, and privacy concerns. Federated learning (FL) presents itself as a viable solution, facilitating decentralized training on devices and preserving data confidentiality. Despite its potential, FL faces obstacles with fine-tuning efficiency, including complex granularity data, static personalized prompt generation, and high energy consumption. This paper introduces a cross-granular and personalized prompt-based green federated tuning (CP²GFed) approach, aiming to address these issues by enabling giant model deployment on devices. CP²GFed introduces a cross-granularity knowledge transfer mechanism to leverage semantic relationships across varying data granularities. Meanwhile, it pioneers in generating dynamic personalized prompts based on inter-device affinities to improve model performance. In addition, CP²GFed meticulously optimizes energy consumption, including model local learning and interaction, by setting local computing steps and selecting communication devices. Empirical results indicate that CP²GFed elevates accuracy by up to 6.64% on diverse datasets and reduces energy consumption by nearly 60% per unit of accuracy, achieving a superior tradeoff between model performance and energy consumption compared to baselines.