On the Influence of an Iterative Affect Annotation Approach on Inter-Observer and Self-Observer Reliability

Affect detection systems require reliable methods to annotate affective data. Typically, two or more observers independently annotate audio-visual affective data. This approach results in inter-observer reliabilities that can be categorized as fair (Cohen's kappas of approximately .40). In an alternative iterative approach, observers independently annotate small amounts of data, discuss their annotations, and annotate a different sample of data. After a pre-determined reliability threshold is reached, the observers independently annotate the remainder of the data. The effectiveness of the iterative approach was tested in an annotation study where pairs of observers annotated affective video data in nine annotate-discuss iterations. Self-annotations were previously collected on the same data. Mixed effects linear regression models indicated that inter-observer agreement increased (unstandardized coefficient B = .031) across iterations, with agreement in the final iteration reflecting a 64 percent improvement over the first iteration. Follow-up analyses indicated that the improvement was nonlinear in that most of the improvement occurred after the first three iterations (B = .043), after which agreement plateaued (B ≈ 0). There was no notable complementary improvement (B ≈ 0) in self-observer agreement, which was considerably lower than observer-observer agreement. Strengths, limitations, and applications of the iterative affective annotation approach are discussed.