PMTSeg: Prompt-Driven Multimodal Transformer for Task-Adapted Remote Sensing Image Segmentation

Multimodal remote sensing image segmentation (MRSIS) is important for intelligent remote sensing (RS) image interpretation, which encompasses three distinct tasks: semantic segmentation, instance segmentation, and panoptic segmentation. Existing methods typically address individual tasks with specialized models, limiting generalization and real-world applicability. Multitask learning (MTL) approaches have introduced separate task heads to unify tasks, yet we identify two key challenges when directly applying them to MRSIS: 1) the modality gap, arising from semantic discrepancies and granularity discrepancies across RS modalities and 2) the task gap, due to varying preferences in learning different segmentation tasks. To overcome these challenges, we propose PMTSeg, a novel prompt-driven multimodal Transformer for task-adapted MRSIS. PMTSeg integrates three key components: 1) task-common multimodal affinity approximation (TMAA); 2) task-common multiscale semantic fusion (TMSF); and 3) a unified prompt-driven segmentation head (PSH). First, TMAA addresses the modality gap by approximating intermodal affinity matrices, extracting task-common features across modalities, and aligning semantic information. Then, TMSF further integrates these features using the scale-matched fusion (SF) at multiple scales to produce enriched, multiscale task-common features. Moreover, to address the task gap, the PSH leverages task-adapted text prompts and task-adapted contrastive loss to model relationships across tasks, enabling adaptive optimization for robust and universal MRSIS performance. Extensive experiments on three MRSIS datasets—VALID, SEMCITY TOULOUSE, and UBCV2—demonstrate that PMTSeg significantly surpasses state-of-the-art methods in all three segmentation tasks, offering a unified and accurate solution to MRSIS.