Unsupervised Domain Adaptation for VHR Urban Scene Segmentation via Prompted Foundation Model-Based Hybrid Training Joint-Optimized Network

Unsupervised domain adaptation for remote sensing semantic segmentation (UDA-RSSeg) is to adapt a model trained on the source-domain data to the target-domain samples, thereby minimizing the need for annotated data across diverse remote sensing (RS) scenes. In urban planning and monitoring, the task of UDA-RSSeg on very-high-resolution (VHR) images has garnered significant research interest. While recent deep learning techniques have demonstrated huge success in tackling the UDA-RSSeg task for VHR urban scenes, a persistent challenge in addressing the domain shift issue remains. Specifically, there are two primary problems: 1) severe inconsistencies in feature representation across diverse domains, characterized by notably differing data distributions; and 2) the domain gap problem due to the representation bias of the source-domain patterns when translating features to predictive logits. To solve these problems, we propose a prompted foundation model-based hybrid training joint-optimized network (PFM-JONet) for UDA-RSSeg on the VHR urban scene. Our approach integrates the notable “segment anything model” (SAM) as a prompted foundation model to leverage its robust generalized representation capabilities, thereby alleviating feature inconsistencies. Based on the feature extracted by the SAM-Encoder, we introduce a mapping decoder (MD) designed to convert SAM-Encoder features into predictive logits. Additionally, a prompted segmentor (PS) is employed to generate class-agnostic maps, which guide the MD’s feature representations. To efficiently optimize the entire network in an end-to-end manner, we design a hybrid training scheme that integrates feature-level and logits-level adversarial training strategies alongside a self-training mechanism. This scheme enhances the model from diverse, compatible perspectives. To evaluate the performance of our proposed PFM-JONet, we conduct extensive experiments on urban scene benchmark datasets, including ISPRS (Potsdam/Vaihingen) and CITY-OSM (Paris/Chicago). On the ISPRS dataset, PFM-JONet surpasses previous SOTA methods by 1.60% in mean IoU value across four adaptation tasks. For CITY-OSM’s adaptation task, it outperforms SOTA by 4.84% in the mean IoU value. These results demonstrate the effectiveness of our method. Furthermore, visualization and analysis reinforce the method’s interpretability. The code of this article is available at https://github.com/CV-ShuchangLyu/PFM-JONet