Prototype-guided Vision Foundation Models fine-tuning for Domain Generalized Semantic Segmentation

Due to the domain shift problem caused by the discrepancy in data distributions between source and target domains, the generalization ability of traditional semantic segmentation models is limited in practical applications. To address this challenge, Domain Generalized Semantic Segmentation (DGSS) aims to learn solely from the source domain while being able to predict pixel-level semantic labels in unseen target domains. Given the rapid development of pre-trained vision foundation models (VFMs) in recent years, we propose a prototype-guided VFM fine-tuning strategy for DGSS. This approach adopts a prototype perspective to selectively update or freeze certain parameters of the pre-trained VFM, enhancing its generalization performance in the field of semantic segmentation. Furthermore, a Query Enhancement Module (QEM) is employed to refine the prototypes, learning invariant representations of the prototypes to further improve the model’s generalization. Extensive experiments demonstrate the superior generalization performance of our method across multiple datasets.