Learning to Evaluate Performance of Multimodal Semantic Localization

Semantic localization (SeLo) refers to the task of obtaining the most relevant locations in large-scale remote sensing (RS) images using semantic information, such as text. As an emerging task based on cross-modal retrieval, SeLo achieves semantic-level retrieval with only caption-level annotation, which demonstrates its great potential in unifying downstream tasks. Although SeLo has been carried out successively, there is currently no work that systematically explores and analyzes this urgent direction. In this article, we thoroughly study this field and provide a complete benchmark in terms of metrics and test data to advance the SeLo task. First, based on the characteristics of this task, we propose multiple discriminative evaluation metrics to quantify the performance of the SeLo task. The devised significant area proportion, attention shift distance, and discrete attention distance are utilized to evaluate the generated SeLo map from the pixel level and the region level. Next, to provide standard evaluation data for the SeLo task, we contribute a diverse, multisemantic, multiobjective SeLo testset (AIR-SLT). AIR-SLT consists of 22 large-scale RS images and 59 test cases with different semantics, which aims to provide comprehensive evaluations for retrieval models. Finally, we analyze the SeLo performance of RS cross-modal retrieval models in detail, explore the impact of different variables on this task, and provide a complete benchmark for the SeLo task. We have also established a new paradigm for RS referring expression comprehension (REC) and demonstrated the great advantage of SeLo in semantics by combining it with tasks such as detection and road extraction. The proposed evaluation metrics, SeLo testsets, and corresponding scripts have been open to access at
https://github.com/xiaoyuan1996/SemanticLocalizationMetrics
.