Confidence Estimation Based on the Explanations of Model’s Predictions

Estimating the confidence of a machine learning (ML) model plays an important role in minimizing undesirable outcomes and safety risks when using ML models in the real world, especially in high-stake application scenarios. However, the majority of existing confidence estimation methods do not work well without incorporating internal information from the model, such as the gradients, internal representations or checkpoints of its training process, which might compromise the security of the model. To alleviate such limitations, we intend to design a reliable confidence estimation method that requires as little internal model information as possible, yet precisely detecting uncertain samples for the model. In this work, we propose a confidence estimation approach that learns the confidence criterion (that is, the true class probability, TCP) based on the explanations of model’s predictions. The proposed approach utilizes the knowledge learned in the model with the leverage of the explanations for model’s prediction rather than directly extracting internal information of the model, so that it can be adopted in a wide range of applications. Several experiments, including experiments on different datasets, different models and non-IID (independent and identically distributed) test datasets, are conducted to examine the effectiveness of our confidence estimation method. The experimental results show that our proposed approach provides a reliable confidence estimation and generally performs better than the compared methods, especially in the capability of recognizing when a ML model makes errors.