SAG-Net: Spectrum Adaptive Gate Network for Learning Feature Representation From Multispectral Imagery

Feature representation plays a key role in matching keypoints, especially for the multispectral images of large spectral difference. On such image pairs, existing methods typically use the two images only, but it is challenging to directly learn spectrum-invariant feature representation due to the complex nonlinear distortion between them. To address this issue, this letter proposes using intermediate-band images to facilitate learning spectrum-invariant feature representation. For this purpose, this work designs a spectrum adaptive gate network (SAG-Net) that consists of a SPectral gate (SPeG) module and a deep feature extractor. The SPeG module selectively activates the spectrum-invariant features according to input image content on-the-fly. It hence allows for training on the images of over two bands simultaneously with a single network without the need of an individual branch per band. To investigate the SPeG module, we also constructed a Landsat 9 Multi-Spectral Images (L9-MSI) dataset including 3167 scenes of aligned images across five spectral bands (visible, B5, B6, B7, and B10) from the Landsat 9 imagery. The experimental results demonstrate the SPeG module can learn common feature representation for varying-band images, and the intermediate B5, B6, and B7 images are useful for the SAG-Net to learn the common feature between visible and B10. On the L9-MSI dataset, the SAG-Net significantly improved the number of correct matches and the matching score (MS). Our dataset will be released at https://github.com/bohanlee/L9MSI-Dataset.git.