Using Publications and Domain Knowledge to Build Research Profiles: An Application in Automatic Reviewer Assignment

Peer-review has been a common practice for quality control in scholarly publications for decades. The ubiquity of the Internet and, subsequently, the availability of easy-to-use Web-based systems (both free and commercial) has made the peer-review process fast, cost-effective and convenient. In a typical scenario, authors upload papers online and manually assign topic-areas; reviewers also sign up by letting the system know about their area of expertise. A rudimentary Paper-Reviewer matching is usually performed by the system and validated by the Program-Chair (for conferences) or by the Editor-in-Chief (for journals). As argued in relevant literature, the peer-review process suffers from several flaws including author's or reviewer's bias in choosing topic-areas and expertise, as well as inter-reviewer agreement, etc. In this research, we explore automatic reviewer assignment for papers by solely considering the content of the papers and the true profile of the reviewers. In this research, we experimented with three approaches to calculate paper-reviewer relevance using the Vector Space Model. We used a set of 10 papers, 30 reviewers and the real paper-reviewer assignment information from a real-conference; and justifed the result of automatic paper-reviewer assignment based on the above three approaches. We noticed that the overlap between real-assignment and automatic-assignment is poor (with only 55-66% of the reviewers being in common). Such a result was not surprising to us, since we are aware that reviewers often express their frustrations claiming that some papers assigned them are not in line with their preferences and expertise. The data-set we used was rather small and suffered from data-sparseness problem and therefore we tried to analyze the automatic-assignment rationales through unbiased human judgment to identify the effect of the above-mentioned approaches in automatic reviewer assignment. We concluded that combining domain-knowledge with automatically extracted keywords (i.e., ontology-driven topic inference using automatically-extracted keywords) could potentially identify the most relevant candidate-reviewers for a paper.