Causal Inference for Leveraging Image-Text Matching Bias in Multi-Modal Fake News Detection

Multi-modal fake news detection has drawn considerable attention with the development of online social media. Existing methods primarily conduct direct cross-modal fusion, while ignoring the image-text matching degree which may introduce unexpected bias. This work studies an unexplored problem in multi-modal fake news detection – how to deconfound and leverage the image-text matching bias to improve the performance of fake news detection. The key lies in two aspects: how to remove the confounding effect of the image-text matching bias during training, and how to utilize the bias in the inference stage since the news with mismatched image and text is more likely to be fake. To achieve our goal, we formulate the fake news detection task as a causal graph that reflects the cause-effect factors, and propose a novel framework –
C
ausal Inference for
L
everaging
I
mage-text
M
atching
B
ias (
CLIMB
) in multi-modal fake news detection. To our best knowledge, this is the first work that considers the image-text matching degree into the fake news detection task with the approach of causal inference. CLIMB can be applied to any fake news detection models with visual and textual features as inputs. Extensive experiments on two real-world datasets validate the effectiveness of CLIMB.