Can Cooke’s model sift out better experts and produce well-calibrated aggregated probabilities?

In decision and risk analysis, Cooke’s classical model is considered one of the most widely used methods for aggregating experts’ probability estimates. However, this model’s average-probability scoring rule may enable experts who dishonestly report their quantile estimates to obtain higher scores and, hence, to receive greater weights. In this study, we adopt the leave-one-out cross-validation technique to perform an out-of-sample comparison of Cooke’s classical model, the equal-weight linear pooling method, and the best-expert approach. Our results indicate that while the performance of the classical model is much poorer after using an out-of-sample analysis, but Cooke’s performance-weight aggregation scheme still significantly outperforms the equal-weight linear pooling method or the best-expert approach. However, the equal-weight approach is more robust than the classical model on the whole.