Spectral-Spatial Attention Fusion Network for Semantic Segmentation of Optical and SAR Image

Semantic segmentation of remote sensing images is a fundamental task in earth observation, and the fusion of multimodal information has been proven to be beneficial for improving segmentation accuracy. However, optical and Synthetic Aperture Radar (SAR) images, as the two most widely used remote sensing data, have great semantic differences due to their individual imaging mechanisms. How to effectively utilize optical and SAR images for joint semantic segmentation is a challenging issue. Therefore, we have proposed a novel spectral-spatial attention fusion network (SSAFNet) for multimodal semantic segmentation, which consists of three main components: the two-stream encoder-decoder network, spectral-spatial attention fusion module (SSAFM), and semantic segmentation head. Firstly, optical and SAR images were input into two independent encoder-decoder streams in parallel to obtain high-level semantic features of different modalities. Then, the features were fed into the SSAFM for deep fusion and thus into the segmentation head to obtain joint optical and SAR image segmentation results. Finally, we conducted comparative experiments on an optical-SAR paired dataset, demonstrating that the joint application of optical and SAR images can improve the accuracy of single-mode semantic segmentation, and the proposed network proved to be progressive by comparison with other methods.