Automatic Identification of Narratives: Evaluation Framework, Annotation Methodology, and Dataset Creation

One of the fundamental components of understanding online discourse in social networks is the identification of narratives. For example, the analysis of disinformation campaigns requires some inference about their communication goals that, in turn, requires the identification of the narratives that they promote. The research in this task involves a number of challenges such as the limited availability of labelled datasets, the subjectivity of the annotators and the time cost of annotation. This article present a definition of the Narrative Identification task, proposes an evaluation framework for Narrative Identification, and a methodology for the creation and annotation of Narrative Identification datasets taking into account the subjectivity of the task. Keeping in mind the goal of comparing systems performance, we explore how to reduce the annotation time while maintaining the reliability of the evaluation. Following this methodology, a set of eight tasks for narrative identification in the political domain has been developed in Spanish and English. Finally, we validated the evaluation framework by analysing its application to DIPROMATS 2024 shared task, together with the performance analysis of baseline and participant systems.