DeepASPeer: Towards an Aspect-level Sentiment Controllable Framework for Decision Prediction from Academic Peer Reviews

Peer review is the widely accepted mechanism to determine the quality of scientific work. Even though peer-reviewing has been an integral part of academia since the 1600s, it frequently receives criticism for the lack of transparency and consistency. Even for humans, predicting the peer review outcome is a challenging task as there are many dimensions and human factors involved. However, Artificial Intelligence (AI) techniques can assist the editor/chair anticipate the final decision based on the reviews from the human reviewers. Peer review texts reflect the reviewers’ opinions/ sentiments on various aspects (e.g., novelty, substance, soundness, etc.) of the paper concerning the research in the paper, which may be valuable to predict a manuscript’s acceptance or rejection. The exact types and number of aspects could vary from one to the other venue (i.e., the conferences or journals). Peer review texts, however, which often contain rich sentiment information about the reviewers and, therefore, their overall opinion of the paper’s research, can be useful in predicting a manuscript’s acceptance or rejection. Here in this work, we study how we could take advantage of aspects and their corresponding sentiment to build a generic controllable system to assist the editor/chair in determining the outcome based on the reviews of a paper to make better editorial decisions. Our proposed deep neural architecture considers three information channels, including reviews, review aspect category, and its sentiment, to predict the final decision. Experimental results show that our model can achieve up to 76.67% accuracy on the ASAP-Review dataset (Aspect-enhanced Peer Review) consisting of ICLR and NIPS reviews considering the sentiment of the reviews. Empirical results also show an improvement of around 3.3 points while aspect information is added to the sentiment information
¹
.
¹
We make our code publicly available athttps://github.com/sandeep82945/-PEERREVIEW-DECISION-Public.gitCCS CONCEPTS• Computing methodologies → Information extraction; • Information systems → Information extraction.