MDFNet: Multimodal Feature Decomposition and Fusion Network for Multimodal Remote Sensing Image Semantic Segmentation

The semantic segmentation of multimodal remote sensing (RS) images, utilizing optical and synthetic aperture radar (SAR) data, has raised attention in recent studies. Advanced studies adaptively modeled and fused modality-share and modality-specific information beneficial for segmentation and achieved competitive performance. However, in challenging scenarios such as low image contrast or blurred textures in optical images and speckle noises or foreshortening in SAR images, modality-specific information may lost. This study proposed a multimodal feature decomposition and fusion network (MDFNet) designed for multimodal RS image semantic segmentation using optical and SAR images. By decomposing multimodal features into modality-share and modality-difference features and applying gradient descent on modality-difference features, modality-specific information beneficial for segmentation can be retained. Specifically, we designed a MDF decoder with MDF blocks. MDF block maps multimodal features into the same feature space and calculates the difference of multimodal features to obtain modality-share and modality-difference features respectively, then fuses these features. MDF decoder optimizes modality-difference features by gradient descent using adaptive modeling, thereby retaining modality-specific information that is beneficial for segmentation. Comprehensive experiments conducted on the DFC20 dataset demonstrated that the proposed MDFNet surpasses representative methods.