Measurement of Rater Consistency by Chance-Corrected Agreement Coefficients

Measurement of consistency in the decisions made by observers or raters is an important problem in clinical medicine. Chance corrected agreement coefficients such as the Cohen and Fleiss Kappas are commonly used for this purpose, though the way that they estimate the probability of agreement 'by chance' has been strongly questioned. Alternatives have been proposed, such as the Aickin Alpha coefficient and the Gwet AC
₁
and AC
₂
coefficients which are gaining currency. A well known paradox illustrates deficiencies of the Kappa coefficients which, it is claimed, are remedied by an approach which grades the subjects according to their probability of being hard to score. The AC
₁
and AC
₂
coefficients result from the application of this grading to the Brennan-Prediger coefficient which may be considered a simplified form of Kappa. This paper questions the rationale of the hardness probability used by AC
₁
and proposes an alternative approach that may be applied to weighted and unweighted multi-rater Cohen and Fleiss Kappas and also Intra-Class Correlation (ICC) coefficients.