Designing argumentation in peer review: Patterns of evaluation in the AltCHI open peer review forum

We examine patterns of evaluation on the open peer review forum of the AltCHI panel of the ACM SIGCHI Conference on Human Factors in Computing Systems, 2012-2013. By analyzing a dataset including all available reviews posted on the AltCHI online platforms, we find that author reviewers are more critical than external commentators, and later submission reviews are more favorable than earlier ones. We find substantial inter-reviewer agreement, partly explainable through persuasive convergence and anchoring effects. Final article acceptance through juried selection is strongly correlated with the average grade in open review, while the number of article reviews is less consequential for selection. The communication design of AltCHI open review balances several conflicting participation incentives, in order to support juried selection for atypical and potentially controversial papers.