ADMH-ER: Adaptive Denoising Multi-Modal Hybrid for Entity Resolution

Multi-Modal Knowledge Graphs (MMKGs), comprising relational triples and related multi-modal data (e.g., text and images), usually suffer from the problems of low coverage and incompleteness. To mitigate this, existing studies introduce a fundamental MMKG fusion task, i.e., Multi-Modal Entity Alignment (MMEA) that identifies equivalent entities across multiple MMKGs. Despite MMEA’s significant advancements, effectively integrating MMKGs remains challenging, mainly stemming from two core limitations: 1) entity ambiguity, where real-world entities across different MMKGs may possess multiple corresponding counterparts or alternative identities; and 2) severe noise within multi-modal data. To tackle these limitations, a new task MMER (Multi-Modal Entity Resolution), which expands the scope of MMEA to encompass entity ambiguity, is introduced. To tackle this task effectively, we develop a novel model ADMH-ER (Adaptive Denoising Multi-modal Hybrid for Entity Resolution) that incorporates several crucial modules: 1) multi-modal knowledge encoders, which are crafted to obtain entity representations based on multi-modal data sources; 2) an adaptive denoising multi-modal hybrid module that is designed to tackle challenges including noise interference, multi-modal heterogeneity, and semantic irrelevance across modalities; and 3) a hierarchical multi-objective learning strategy, which is proposed to ensure diverse convergence capabilities among different learning objectives. Experimental results demonstrate that ADMH-ER outperforms state-of-the-art methods.

I. Introduction

Knowledge Graphs (KGs) are structured semantic networks that have been widely adopted in various applications, including semantic search, recommendation systems, and natural language question answering [1]. Nevertheless, traditional KGs represent entities and relations using triples, which weakens the ability of machines to fully describe and understand the complexity of the real world [2]. To overcome this issue, Multi-Modal Knowledge Graphs (MMKGs) like MMKG [3] and Richpedia [4] have been developed in recent years. These MMKGs enrich the knowledge diversity by incorporating additional multi-modal knowledge such as text and images into traditional KGs. Although MMKGs contain abundant information, they still suffer from the problems of low coverage and incompleteness, which has significantly hindered their applications [1].