Consistency Analysis of NLP Approaches for a Conference Reviewer-Manuscript Match-Making System

Peer-review process is an important part of scholarly communication. The quality of a conference also depends on its peer-review process. The selection of a competent reviewer to review a submitted manuscript in a conference, is a crucial facet of a peer-review process. This selection relies on the adopted match-making approach along with the constraint optimization reviewer allocation algorithm. The match-making approach needs to be consistent with its decision of selection and allocation of the reviewers. In this work, we proposed a framework for evaluating the consistency of various standard NLP approaches that are used for match-making process in a conference. The consistency analysis was performed over a real multi-tracked conference organized in 2019. We showed that the Contextual Neural Topic Modeling (CNTM) with word embedding technique was most consistence among all the 13 approaches that we chose to analyze.