Image-Text Matching With Shared Semantic Concepts

Image-text matching is a fundamental vision-language task with massive applications. It aims to retrieve textual descriptions directly from query images and vice versa. However, textual semantic concepts, including objects, actions, and scenes, are apparent and structured. Directly correlating them with pixel-level visual representations hinders the ability of the retrieval model to reason about high-level relationships between two modalities. Therefore, it is challenging to align complex visual information with ambiguous descriptions. To tackle this issue, a shared semantic concepts fusion network is proposed to identify potential semantic relationships between image and text, and to narrow the cross-modal gap. Specifically, the semantic concepts are extracted in a shared space by a plug-and-play network that exploits inner knowledge and latent relationships. Afterwards, the visual and textual semantic concepts and enhanced representations are incorporated by a multi-feature fusion module. Experimental results are conducted on two benchmark datasets, and our model outperforms the state-of-the-art methods by 0.8% and 1.4% on R@1 for text and image retrieval of Flickr30K.