Explainable Multimodal Fake Posts Detection Using Feature Extraction with Attention Mechanisms

Social media has become one of the most critical infrastructures for busy modern people to obtain information efficiently. Despite being a major source of information, as users become more diverse, low-literacy users believe incorrect information and repeatedly transmit wrong content. In addition, recent developments in generative AI technology have made it easy for anyone to generate large amounts of text and images that make others believe, but many of these are mistakes or plagiarism, adversely affecting a more significant number of people and a wider area. In fake post-detection tasks, there is generally a trade-off between identification performance and model transparency explaining the basis for identification. However, with the advent of the attention mechanism, it is now possible to incorporate explanatory mechanisms into opaque models such as neural networks, increasing the possibility of pursuing identification performance and explainability. Thus, we propose combining an opaque but suitable model for multimodal with methods directing attention to essential elements and extracting features of AI -generated content. The proposed method is able to detect fake posts better than conventional methods in multimodal data and can provide clues for their detection.