Learning Distinctive Feature Representation for Descriptors on Multimodal Images by Incorporating Multiple Negative Samples

This article proposes a method composed of a loss function and a feature extractor structure, to learn the distinctive feature representation (DIFR) for descriptors on multimodal images. The distinctiveness of the feature representation plays a key role in matching keypoints, and has drawn a lot of research attention. In the past, many methods were proposed that either employed handcrafted features or deep neural networks to learn the feature representation. However, the incorrect ratio of built keypoint matches still remains high due to the insufficient distinctiveness of the associated descriptors. To address this problem, this work designs a new contrastive loss function that for the first time incorporates multiple negative descriptors from both the reference image and the test image, to constrain the networks to learn the DIFR for descriptors. To further enable the descriptors to encode more global information, we design RestExtractor, a novel CNN-Transformer structure that effectively adapts the self-attention mechanism to the feature learning for multimodal images. In the RestExtractor, we design a gated-v block and a receptive field enhanced feed-forward network for extracting distinctive modal-shared feature representation. We evaluated the proposed method on two challenging multimodal image datasets, VIS-SAR and VIS-IR. The experimental results showed that the proposed DIFR significantly improved the number of correct keypoint matches over the state-of-the-art methods.