Effective Diversity Optimizations for High Accuracy Deep Ensembles

Deep Neural Network Ensembles (Deep Ensembles) have emerged as a popular technique for enhancing overall prediction accuracy by leveraging the complementary predictive capabilities of multiple diverse models. Ensemble member models with high diversity tend to have high independence in making prediction errors, which hold the potential to improve overall ensemble predictive performance. However, it still remains an open challenge to precisely measure the ensemble diversity to guide the selection of high-quality deep ensembles. This paper presents a novel Synergistic Diversity (SQ) framework to optimize ensemble diversity measurements, which can efficiently select high-quality deep ensembles with high ensemble accuracy. First, we show that our SQ metric can significantly enhance ensemble diversity measurements by considering the synergistic effects of both ensemble member models and different diversity measures, effectively capturing the complementary predictive capabilities of the member models in an ensemble team. Second, our SQ metric exhibits a strong correlation with both ensemble accuracy and ensemble robustness against Out-Of-Distribution (OOD) samples, which allows it to efficiently identify the top-K high-quality deep ensembles with high accuracy and robustness. Third, systematic experiments are performed on several benchmark datasets and demonstrate that our SQ metric can effectively optimize ensemble diversity measurements and achieve high ensemble accuracy for deep neural network ensembles and Large Language Model (LLM) ensembles.