Knowledge-Enhancement Module for RGB-T Semantic Segmentation in Remote Sensing

In accomplishing the task of semantic segmentation of RGB-T remote sensing images, there is a great challenge due to severe occlusion, long-tailed data distribution, and insufficient pixel representation of certain objects. The effective use of high-level semantic contexts among various ground object categories is crucial, yet presents considerable diffi-culties. Traditional RGB-T remote sensing image semantic segmentation methods often fail to capture and utilize complex relationships and interdependencies among these categories, leading to reduced semantic segmentation accuracy. This paper proposes a novel, knowledge-enhancement module, empowering the model to utilize human-like commonsense knowledge. Specifically, we firstly employ the self-attention and cross-attention mechanisms to fuse RGB and Thermal features. Subsequently, we collect the weights of the classification layer to get a high-level semantic pool based on all the categories. Alongside this, a prior knowledge graph is developed to enable information propagation among all categories. We applied this knowledge-enhancement module to enhance the Mask R-CNN, named KEMask R-CNN. Experiments results on the RS RGB-T dataset demonstrate the progressiveness of the proposed method.