Semantics-Consistent Representation Learning for Remote Sensing Image–Voice Retrieval

With the development of earth observation technology, massive amounts of remote sensing (RS) images are acquired. To find useful information from these images, cross-modal RS image–voice retrieval provides a new insight. This article aims to study the task of RS image–voice retrieval so as to search effective information from massive amounts of RS data. Existing methods for RS image–voice retrieval rely primarily on the pairwise relationship to narrow the heterogeneous semantic gap between images and voices. However, apart from the pairwise relationship included in the data sets, the intramodality and nonpaired intermodality relationships should also be considered simultaneously since the semantic consistency among nonpaired representations plays an important role in the RS image–voice retrieval task. Inspired by this, a semantics-consistent representation learning (SCRL) method is proposed for RS image–voice retrieval. The main novelty is that the proposed method takes the pairwise, intramodality, and nonpaired intermodality relationships into account simultaneously, thereby improving the semantic consistency of the learned representations for the RS image–voice retrieval. The proposed SCRL method consists of two main steps: 1) semantics encoding and 2) SCRL. First, an image encoding network is adopted to extract high-level image features with a transfer learning strategy, and a voice encoding network with dilated convolution is devised to obtain high-level voice features. Second, a consistent representation space is conducted by modeling the three kinds of relationships to narrow the heterogeneous semantic gap and learn semantics-consistent representations across two modalities. Extensive experimental results on three challenging RS image–voice data sets, including Sydney, UCM, and RSICD image–voice data sets, show the effectiveness of the proposed method.