Peer Grading Eliciting Truthfulness Based on Autograder

Peer grading has diverse applications in many fields, including the peer grading of open assignments in online courses. The major challenge in peer grading is improving the seriousness (reviewing carefully) of reviewers. Previous studies have proposed several incentive reward mechanisms intended to reward or punish reviewers. Although these mechanisms are effective in massive open online courses environments with a large number of students, they are not suitable for small private online courses (SPOCs) environments with a small number of students. This article analyzes the characteristics of reviewers in a small private online course environment and designs an incentive and reward mechanism based on a hybrid human–machine peer grading framework, including an autograder (reviewed by the machine). The main component of the proposed mechanism is the autograder whose function is to evaluate the reviewers’ reports (reviewers refer to students) and provide feedback information on reports to reviewers. The framework can incentivize reviewers to review carefully and give their reports truthfully when the reports between reviewers’ and the autograder’ are consistent. The proposed framework is verified experimentally. The experimental results demonstrate that the hybrid human–machine peer grading framework can effectively incentivize reviewers to review submissions carefully and improve the overall effect of peer grading.