Transformer Based Image-Text Consistency Analysis for Infographic Articles

We present a multi-modal T5 Transformer-based method for image-text semantic consistency analysis that are targeted at infographic articles. Infographics provide graphical presentations of information and have wide applications in visualization and news media. Many media forensics methods are developed for consistency analysis among texts and images, however methods that work specifically for infographic semantic consistency are still scarce. We integrate the Natural Language Inference (NLI) of the T5 Transformer with scene text extraction for infographic consistency analysis. Experimental study is performed on the DARPA Semantic Forensics (SemaFor) Evaluation dataset for the task of news article inconsistency detection. The test set contains 248 infographic articles that are evenly split between manipulated and pristine samples. For evaluation, low False Positive Rate (FPR) is desired in media forensics. We use the metrics of True Positive Rate (TPR) at low 0.08 FPR, as well as the standard ROC AUC and Equal Error Rate (EER). Results show that our method outperforms mainstream methods from other SemaFor performer teams.