CLMSI: A Novel Image-Text Named Entity Recognition Method Based on Contrast Learning and Multimodal Semantic Interaction

Most existing multimodal named entity recognition (MNER) methods cannot align image and text well, and fail to effectively fuse image-text semantic information, leading to suboptimal MNER accuracy. To this end, this paper proposes a novel named entity recognition model based on Contrastive Learning and Multimodal Semantic Interaction (named CLMSI). Specifically, to narrow the semantic gap between image-text modalities, making semantically similar image-text representations closer and pushing dissimilar ones further apart, we propose a contrastive learning-based cross-modal alignment method. Moreover, to integrate multimodal information effectively and filter out the noise introduced by image features as thoroughly as possible, we propose to incorporate image descriptions as additional features to enrich the multimodal representation and to reduce the semantic bias of the image information. Extensive experiments conducted on two MNER benchmark datasets, Twitter-2015 and Twitter-2017, demonstrate the superiority of the proposed method. Further ablation studies and case analyses also affirm the efficacy of CLMSI.