Semantic-Enhanced Attention Network for Image-Text Matching

Image-text matching is an important task in cross-modal information processing, which consists of evaluating the similarity between images and text. However, the data of the two modalities have different distributions and representations, which cannot be directly compared. Therefore, the two modalities need to be processed separately. Most existing image-text matching methods extract image fragments and text fragments separately, using an attention mechanism to establish relationships between regions in the image and words in the text. Subsequently, they aggregate the similarity of these images and text fragments. Regardless, not all fragment alignment is meaningful, and irrelevant fragment alignment will bring redundancy and reduce retrieval accuracy. In addition to that, there are contextual relationships between image regions and between text words, which should also be taken into account. In this paper, we simultaneously consider these two aspects and propose a Semantic-Enhanced Attention Network (SEAN). It first focuses on each modality, mining contextual relationships between fragments within each modality and aggregating contextual information into visual and textual embeddings. Then, we compute the similarity between all image regions and text fragments, focusing all attention on the region-word pairs with the highest similarity. Finally, we summarize these to infer the similarity between the image and text. Our method achieves competitive results on two generalized image text retrieval datasets, Flickr30K and MS-COCO.