Problems with scoring methods and ordinal scales in risk assessment

Risk assessment methods based on scoring methods that rate the severity of each risk factor on an ordinal scale are widely used and frequently perceived by users to have value. We argue that this perceived benefit is probably illusory in most cases. We begin by describing a number of common scoring methods currently used to assess risk in a variety of different domains. We then review the literature on the use of ordinal scales in risk analysis, the use of “verbal scales” for eliciting estimates of risks and probabilities, and the extensive research about peculiar human errors when assessing risks. We also supplement this overview with some data of our own. When these diverse kinds of evidence are combined, the case against scoring methods is difficult to deny. In addition to the evidence against the value of scoring methods, there is also a lack of good evidence in their favor. We conclude our overview by reviewing the reasons why risk assessment approaches should describe risk in terms of mathematical probabilities.