Boundary-aware Spatial and Frequency Dual-domain Transformer for Remote Sensing Urban Images Segmentation

Semantic segmentation of Remote Sensing (RS) images refers to labeling each pixel with a class to identify objects or land cover types. Existing mainstream spatial-domain semantic segmentation methods are mainly categorized into Convolutional Neural Network (CNN)-based and Vision Transformer (ViT)-based approaches. The former excels at capturing local features, while the latter is adept at extracting global features. Several recent approaches consider combining CNN and ViT to efficiently capture local and global features. However, these approaches still struggle to capture complete features of the RS images, resulting in inaccurate segmentation. To address this issue, we introduce the Fast Fourier Transform (FFT), which transforms images into the frequency domain for feature extraction, acquiring the image-size receptive field that can complement spatial-domain methods. Based on this, we propose a Boundary-aware spatial and frequency Dual-domain Transformer, termed Dual-domain Transformer. Specifically, our Dual-domain Transformer incorporates a Dual-domain Mixer (DualM), where the spatial-domain branch combines depthwise convolution and the attention mechanism to extract local and global features effectively, while the frequency-domain branch uses FFT to extract image-size features. The two branches complement each other, enabling a more comprehensive feature extraction of RS images. Meanwhile, a boundary-guided training strategy utilizing a Boundary-Aware Module (BAM) is devised to constrain the model extract and predict boundary detail texture, which is an auxiliary task. Additionally, the decoder incorporates a Scale-feature Fusion Module (SFM) for adaptive information fusion between the encoder and decoder. Comprehensive experiments on the Zeebrugge and ISPRS datasets, including Vaihingen and Potsdam, showcase that the Dual-domain Transformer significantly outperforms state-of-the-art (SOTA) methods.