Inference assessment on a probability scale

The performance of a probability inference or forecast can be expressed as the average forecasted probability by computing the geometric mean of the forecasted probabilities. This metric is obtained by transforming the information theoretic measure, commonly known as the logarithmic score or the average surprisal, into the probability domain via application of the exponential function. The average probability forecast is composed of two sources of uncertainty, the underlying uncertainty of the data source and the divergence between the model and source. Expressed as probabilities, these three measures of the performance can be plotted as part of the probability calibration curve comparing the forecasted probabilities with the measured source probability. Insight into the under- or over-confidence of the model is obtained by computing the weighted generalized mean of the forecasted and source probabilities. A comparison of the 2016 US Presidential election forecasts is used to illustrate the utility of the method.