Framework for Algorithmic Bias Quantification and its Application to Automated Sleep Scoring

The validation of predictive algorithms is gaining importance with the increasing use of AI. Traditional validation of software seeking, for example, clinical certification involves correlation or Bland-Altman analysis comparing differences between predicted and reference values. However, such approaches are subject to simplifying assumptions on the algorithmic errors: normality of their distribution, homogeneity of variance, and independence from external factors. Our study - motivated by the sleep medicine use-case - proposes an in-depth quantification of systematic algorithmic error (bias) using the flexible statistical tool GAMLSS. Our approach allows the estimation of the bias distribution, identification of bias-generating factors, and extrapolation of various quantities assessing prediction validity.