Incentive Design in Peer Review: Rating and Repeated Endogenous Matching

Peer review (e.g., grading assignments in Massive Open Online Courses (MOOCs), academic paper review) is an effective and scalable method to evaluate the products (e.g., assignments, papers) of a large number of agents when the number of dedicated reviewing experts (e.g., teaching assistants, editors) is limited. Peer review poses two key challenges: 1) identifying the reviewers’ intrinsic capabilities (i.e.,
adverse selection
) and 2) incentivizing the reviewers to exert high effort (i.e.,
moral hazard
). Some works in mechanism design address
pure
adverse selection using
one-shot
matching rules, and
pure
moral hazard was addressed in repeated games with
exogenously given
and
fixed
matching rules. However, in peer review systems exhibiting both adverse selection and moral hazard, one-shot or exogenous matching rules do not link agents’ current behavior with future matches and future payoffs, and as we prove, will induce myopic behavior (i.e., exerting the lowest effort) resulting in the lowest review quality. In this paper, we propose for the first time a solution that
simultaneously
solves adverse selection and moral hazard. Our solution exploits the
repeated
interactions of agents, utilizes
ratings
to summarize agents’ past review quality, and designs matching rules that
endogenously
depend on agents’ ratings. Our proposed matching rules are easy to implement and require no knowledge about agents’ private information (e.g., their benefit and cost functions). Yet, they are effective in guiding the system to an equilibrium where the agents are incentivized to exert high effort and receive ratings that precisely reflect their review quality. Using several illustrative examples, we quantify the significant performance gains obtained by our proposed mechanism as compared to existing one-shot or exogenous matching rules.