Between Acceptance and Rejection: Challenges for an Automatic Peer Review Process

The peer review process is the main academic resource to ensure that science advances and is disseminated. To contribute to this important process, classification models capable of predicting the score of a review text (RSP) and the final decision of a paper (PDP) were created. But what challenges prevent us from having a fully efficient system responsible for these tasks? And how far are we from having an automated system to take care of these two tasks? To answer these questions, in this work, we evaluated the general performance of existing state-of-the-art models for RSP and PDP tasks, and investigated what types of instances these models tend to have difficulty classifying and how impactful they are. We found, for example, that the performance of a model to predict the final decision of a paper is 23.31% lower when it is exposed to difficult instances and that the classifiers make mistake with a very high confidence. These and other results lead us to conclude that there are groups of instances that can negatively impact the model’s performance. That way, the current state-of-the-art models have potential to helping editors to decide whether to approve or reject a paper, however we are still far from having a system that is fully responsible for scoring a paper and decide if it will be accepted or rejected. CCS CONCEPTS • Computing methodologies $\rightarrow$ Natural language processing; Supervised learning by classification.