Iterative processing algorithm to detect biases in assessments

In order to assess a large number of diverse projects as fairly and rapidly as possible, a procedure often adopted is to use a panel consisting of a large number of experts, only a small number of whom assess each project. Since no one expert assesses all the projects, conscious or unconscious bias regarding overall standards by any expert will advantage or disadvantage the projects assessed by that particular expert. This paper presents an iterative algorithm that has been used successfully to detect and correct for such biases. Each expert's assessments are modeled as differing from the ideal as a result of a shift of mean and having a standard deviation that is too low or too high. This model is used in conjunction with the concept of "paired assessments" to account for individual projects being of unusually high or low quality and so to evaluate the discrepancy from the ideal marks. The same computer program also has applications in the peer-review or expert-evaluation of research proposals, and any other situation involving subjective assessments by a restricted number of persons.