A Multimodal Feature Fusion Network for Building Extraction With Very High-Resolution Remote Sensing Image and LiDAR Data

Building extraction from remote sensing images is extremely important for urban planning, land-cover change analysis, disaster monitoring, and so on. With the growing diversity in building features, shape, and texture, coupled with frequent occurrences of shadowing and occlusion, the use of high-resolution remote sensing image (HRI) alone has limitations in building extraction. Therefore, feature fusion using multisource data has gradually become one of the most popular. However, the unique characteristics and noise issues make it difficult to achieve effective fusion and utilization. Thus, it is very challenging to realize the full fusion of multisource data to achieve complementary advantages. In this article, we propose an end-to-end multimodal feature fusion building extraction network based on segformer, which utilizes the fusion of HRI and light detection and ranging (LiDAR) data to realize the building extraction. First, we utilize the segformer encoder to break through the limitations of the traditional convolutional neural network (CNN) with the restricted receptive field, so as to achieve effective feature extraction of complex buildings. In addition, we propose a cross-modal feature fusion (CMFF) method utilizing the self-attention mechanism to ensure the fusion of multisource data. In the decoder part, we propose a multiscale upsampling decoder (MSUD) strategy to achieve a full fusion of multilevel features. As demonstrated by experiments on three datasets, our model shows better performance than several multisource building extraction and semantic segmentation models. The intersection over union (IoU) for buildings on the three datasets reaches 91.80%, 93.03%, and 84.59%. Subsequent ablation experiments further validate the effectiveness of each strategy.