Learning Rotation-invariant Neighbor Consensus for Mismatch Removal

Mismatch removal aims to identify high-quality correspondences from initial matches, and is a critical step in the image-matching pipeline. Although existing learning-based mismatch removal methods achieve promising performance, they are sensitive to geometric transformation, such as image rotations, resulting in poor generalization when applied to remote sensing images. In this paper, we introduce a rotation-invariant neighbor consensus network, termed RI-NCNet, to extract geometric transformation-insensitive graph contexts for accurate mismatch removal in remote sensing scenarios. Specifically, we propose a frame-based neighbor consensus mining (FNCM) module that maps the neighbor pattern of each correspondence into point-specific local reference frames (PLRFs) to counteract image rotation and translation. Instead of mining context consensus directly from input coordinates, FNCM learns to derive rotationally invariant consensus representation from PLRFs-based relative neighbor encodings. This could effectively improve the robustness of RI-NCNet to geometric transformations. Considering that global awareness contributes to the discriminability of local context, we further propose a multiscale aggregation strategy to make the network aware of global context. Experimental results on various real data demonstrate that the proposed RI-NCNet achieves a superior generalization ability to diverse remote sensing scenarios, and outperforms existing state-of-the-art mismatch removal methods.