Multimodal Fake News Prediction using a Two-Transformer Architecture approach with Llama

In today’s era, the spread of fake news is similar to the spread of a forest fire. This is due to the fact that nowadays a vast amount of people use social media. So the probability of misinformation spreading is very high through these online social media handles, and some people even re-post and believe that without even checking the facts. In this project, a multimodal dataset called Weibo is used. The approach includes the creation of image and text embeddings, which are then passed through a two-transformer network, after which the decoder outputs are fused and fed into the Llama model for classification. Data augmentation is employed for the generation of diverse training samples, along with the necessary preprocessing. The results yielded by the model proposed in this research highlights a significant refinement with respect to the traditional ones. Hence, this solution can combat the spread of fake news to some extent.