On the False Notion of Misclassification Probability as Uncertainty in Machine Learning-Assisted Measurements

Artificial Intelligence, including Machine Learning, is now a prevalent tool for many measurement systems. Like any measurement, the uncertainty of machine learning-assisted measurement is an integral part of measurement and must be quantified. On the surface, the notion of uncertainty, as defined by relevant measurement standards, appears to be inversely similar to the notion of classification confidence, as used by the machine learning community. In other words, machine learning’s misclassification probability is sometimes taken to represent the uncertainty of the measurement it contributes to. While this notion is frequently used in the existing literature, it is a false notion. It has been argued and shown that misclassification probability and uncertainty are not the same thing. In this paper, we show empirically that classification accuracy is adversely impacted by this false notion significantly–by more than 17% on average and by as much as 38% in the worst case in the two datasets we tested. This empirical observation confirms that misclassification probability should not be substituted for uncertainty.