A Hierarchical and Multi-modal Framework for Place Recognition with a Learnable Metric

The descriptive power and comparative metric of features are the key factors in multi-modal place recognition tasks. However, learning a multi-modal descriptor directly for comparison may encounter dominating modality problem, and the typical Euclidean/cosine distances-based metric may lack of comparative ability with inadequate distinction on features. To address these problems, this article presents a hierarchical and multi-modal place recognition framework, named HMPR, that utilizes both global and local features extracted from image and point cloud to retrieve place matches. To be specific, visual global features are used in the first stage to obtain an initial list of top-K candidates, then reranking is done by leveraging multi-modal local features in the refinement stage. Instead of learning a multi-modal descriptor vector, we define a learnable metric in the refinement stage to generate similarity scores between the query and the candidate references by leveraging multi-modal feature fusion network. For each modality in the network, cross-attention between query and candidate references is utilized to enhance the local features. These features are then used to generate modality-specific attention weights for a later modality fusion, followed by a sigmoid function to generate similarity scores. Experimental results on public datasets show that our proposed hierarchical and multi-modal framework achieves comparable performance and efficiency compared with several state-of-the-art methods on place recognition task and has well generalization ability. Our code is publicly available at https://github.com/hhuscf/HMPR.