MESAC: Learning to Remove Mismatches via Maximizing the Expected Score of Sample Consensuses

Most learning-based methods require labelling the training data, which is time-consuming and gives rise to wrong labels. To address the labelling issues thoroughly, we propose an unsupervised learning framework to remove mismatches by maximizing the expected score of sample consensuses (MESAC). The proposed MESAC can train various permutation invariant networks (PINs) based on training data with no labels, and has three distinct merits: 1) the framework can train various PINs in an unsupervised mode such that these are immune to wrong labels; 2) the gradients of the expected score are explicitly calculated by a revised score-function estimator, which can avoid gradient explosion; 3) the distribution of matching probabilities is learned from the PIN and precisely modelled by a categorical distribution, which can decrease the sampling times and improve the computational efficiency accordingly. Experiments of testing datasets disclose that mean recall is increased by at most 77% when pure PINs are embedded in MESAC, and mean precision is also improved by 16%. Applications in pose recovery indicate that the success rates of MESAC-integrated PINs outperform the compared methods when training with neither matching labels nor ground truth epipolar geometry (EG) constraints, showing the great potential of MESAC in mismatch removal.