Unintentional affective priming during labeling may bias labels

Online platforms displaying long streams of examples are often employed to gather labels from both experts and crowd workers. While previous work in crowdsourcing focused on objective tasks and estimating error parameters of annotators, collecting labels in a subjective setting (e.g. emotion recognition) is more complicated due to different interpretations of examples. These interpretations could be influenced by many factors such as annotator mood and previously seen examples. In this work, we examine two hypotheses of order-dependent biases in sequential labeling tasks: negatively auto-correlated sequential decision making and positively auto-correlated affective priming. Using controlled generation of facial expressions, we find that i) annotators achieve higher agreement when presented examples in the same sequential order, ii) the valence label of the current image positively correlates with the previous labels given. While we also observe a positive correlation between labels and the number of preceding positive and negative images seen, this correlation is highly dependent on example ordering. Our findings demonstrate that randomized examples given to annotators may produce systematic bias in labels. Future data collection should present examples in orderings which mitigate such bias.