Evidence-Claim Relevance Decoupling for Multimodal Fact-Checking

With the proliferation of misinformation on social media, the need for fact-checking has increased, garnering the attention of researchers. The main task of fact-checking is to analyze the semantic relationship and logic between the evidence and the claim, so as to determine the veracity of the claim. The semantic relationship between the claim and the evidence can be strong or weak. However, previous studies ignores the disparities in the strength of semantic relationship between evidence and claims in the process of evidence modeling for fact-checking, neglects the role of weak semantically related evidence in verifying claims. Weakly semantically related evidence logically determines the veracity of the claim sometimes. This paper proposes an evidence-claim relevance decoupling (ECRD) framework for multimodal fact-checking, where the claim and the evidence could be separated by decoupling the strong and weak relevant semantic parts in multimodal dual-stream models. Specifically, we first project text and image modalities into two distinct spaces. One space focuses on tightly coupled semantics between the claim and evidence, reducing the gap. Another space explores the loosely coupled parts, analyzing the various meaning between claim and evidence to reveal potential underlying connections. Then, the fully connected visual and textual graphs are constructed for above two spaces. Finally, the responses obtained by interacting in above graphs are fed into the fusion layer. On FACTIFY dataset, which features diverse text-image relationships, and MultiFact dataset, experimental results show that the proposed ECRD framework could improve the performance of classification significantly compared to baseline models and produces a competitive performance.