Leveraging Rationality Labels for Explainable Claim Check-Worthiness

Determining the check-worthiness of claims is an integral process in fact-checking. Despite ongoing advancements in claim check-worthiness detection, there remains a notable gap of transparency in elucidating why a claim is deemed check-worthy. Addressing this gap, we introduce an explainable approach to claim check-worthiness by incorporating ‘rationality labels.’ Closely mirroring the complex decision-making process of human cognition, these labels evaluate claims on multiple aspects, including factual verifiability, societal impact, and the likelihood of causing public unrest. Unlike existing systems that provide simplistic binary judgments on check-worthiness, our framework, CheckMate, delves into the nuanced human-like reasoning behind the assessment of claim check-worthiness for fact-checking. We also introduce CheckIt, the first claim check-worthiness dataset of 5920 tweets with rationality labels. This multi-dimensional approach offers a significant advancement over traditional binary evaluation systems. We compare our proposed approach with several baseline systems and comprehensively analyze the results, including quantitative and qualitative comparisons. Our data and code can be found here.