Confusion Matrices: A Unified Theory

The confusion matrix is a key tool for understanding and evaluating models in supervised classification problems. Various matrices are proposed depending on the problem framework: single-label, multi-label, or even soft-label restricted to probability distributions. However, most of these approaches are not compatible with each other and lack theoretical justification. Leveraging optimal transport theory and the principle of maximum entropy, we propose a unique confusion matrix applicable across single, multi, and soft-label contexts. The Transport-based Confusion Matrix (TCM) extends the classic Confusion Matrix (CM), being identical in the single-label context. TCM introduces a comprehensive, theory-supported description of previously inaccessible errors, thereby enhancing the consistency and scope of machine learning evaluation.