An Empirical Investigation of the Evaluators’ Scoring of Vendors’ Responses to an RFP of a Large Healthcare System

Request for Proposal (RFP) is a solicitation of proposals from vendors and they are often judged by human experts from varying backgrounds and experiences. This is typically done because large technical RFPs require a diverse group of evaluators who will bring their skills and experience to bear. However, different people with different backgrounds may evaluate proposals in different ways. In this paper, we examine the variability between expert proposal ratings through a case study, and try to determine how diversity of expertise affects the consistency of scoring results. We evaluate this by using scores of an RFP of a large industrial system by 20 different human experts from six different backgrounds. Our results suggest that there is not always a significant difference among raters in the scores given to vendors. Thus, human experts with varying but related backgrounds may judge in a similar manner and can be used to evaluate the RFP, provided there is a high level agreement between experts and the RFP is created through their consensus. This empirical evidence does not exist in the literature and it is a novel contribution of this paper.