Cross-Modal Semantic Relations Enhancement With Graph Attention Network for Image-Text Matching

Image-text matching is a vital task in multi-modal intelligence. Recently, researchers have moved beyond simply aligning fragments between image regions and text words at a low level. They are now focusing on high-level alignment with semantic relations to enhance performance, and significant progress has been made. However, almost all existing research concentrates on extracting semantic relations between fragments within individual modalities, overlooking the interaction between modalities. This oversight results in a deficiency of high-level semantic and consistency information, ultimately diminishing performance. To address this, we argue that integrating intra-modal and inter-modal semantic relations between fragments could help improve feature quality and promote consistency in semantic space by enabling more comprehensive interactions between modalities. Therefore, we propose a novel Cross-modal Semantic Relations Enhancement Network (CSREN), which designs an implicit semantic relations graph and employs a graph attention network to adaptively mine semantic relations from both modalities, separately and collectively. Quantitative experiments on Flickr30K and MS-COCO demonstrate that our method significantly outperforms existing approaches, achieving improvements of 5.4% to 24.5% for text retrieval and 9.7% to 26.9% for image retrieval.