Collusion in educational peer assessment: How much do we need to worry about it?

Several decades of research have shown peer assessment to be an effective pedagogical approach. Researchers have shown that peer assessment has the potential to provide students more copious, timely and helpful feedback, and also helping reviewers to learn as well. In recent decades, peer assessments in educational settings have increasingly been facilitated by online tools. Some MOOC platforms also rely on aggregated peer-assessment scores to assign grades for each artifact. However, in peer assessment, students can potentially game this process, and thereby harm the validity and reliability of the aggregated scores. Most of instructors assume that the majority of the students' peer assessments are honest since most of the peer assessment is done in double-blind fashion. This assumption only holds in the absence of organized collusion - when no more than a small number of students game the peer assessment and give each other very high scores. This paper identifies two types of collusion that we have observed. They are small-circle collusion and pervasive collusion. Small-circle collusion refers to the behaviors of students who form small circles and give higher peer review grades to each other. Pervasive collusion refers to students assigning top grades to all the submissions they review. We also present our algorithms for detecting these two types of colluders. Our experiments are based on a peer-assessment dataset shared by multiple peer-assessment systems. By removing these colluders' peer assessments, we are able to estimate how much inflation is brought by colluders in educational peer assessment.