Multi-Modal Relation Extraction Enhanced by Out-of-Text Visual Information

Multi-modal relation extraction (MRE) aims to extract semantic relations between two textual entities with the help of visual information. Existing studies typically leverage visual information by directly detecting objects in images or retrieving additional information based on the target text or image. However, we demonstrate through extensive experiments that the key to performance improvement brought by most existing methods is not the visual content, but the dataset bias — such as inconsistent entity type annotations or imbalanced counts of visual objects across samples. In other words, the visual modality has not been effectively utilized in existing studies. To address this issue, in this paper, we propose a novel paradigm to efficiently leverage visual information and achieve stable performance gains. Different from existing methods, the core idea of the proposed paradigm is that the introduced visual information should supplement the key knowledge beyond the text. Specifically, we first use prior knowledge that is not explicitly mentioned in the text as key clues to retrieve relevant images that contain out-of-text information. Then, an image filtering mechanism is designed to ensure the diversity and effectiveness of visual information. Finally, we design a multimodal fusion method based on the attention mechanism to accurately extract relationships between entities. Experimental results on widely used datasets show that our proposed method can steadily improve the performance of textual extraction, and outperforms existing baselines.