Active Learning for Practical Misinformation Classification in Social Media: a Case Study on COVID-19

Misinformation on social media has become a significant societal issue, as these platforms increasingly serve as central hubs for human interaction. The recent COVID-19 pandemic vividly illustrated how the rapid spread of misinformation can lead to adverse personal and societal impacts, exacerbated by the ease with which information is generated, shared, and consumed on these platforms. Much of the previous research has focused on using machine learning algorithms to detect and identify misinformation in social media posts, primarily operating within a strict supervised learning framework where annotated datasets containing both accurate and misleading information are available.However, especially with the rise of generative AI, the task of annotating misinformation posts has become increasingly resource-intensive, requiring substantial domain-specific expertise. As a result, many of these algorithms struggle to adapt to the real-world environment, where data distribution and topics are constantly evolving on social media platforms. To bridge this gap, our paper evaluates the effectiveness of active learning approaches for classifying misinformation with a focus on COVID-19. We demonstrate that achieving a highly accurate classifier with a training dataset significantly smaller than those required in previous studies is possible. Furthermore, we introduce a novel uncertainty estimation method for active learning, the Embedding Vector Similarity (EVS) measure, which leverages the latent embedding space of large language models. This metric enhances the performance of active learning for misinformation classification, reducing the dependence on large, high-resource annotated datasets while maintaining the quality of the analysis for addressing this critical social issue.