Enumerating the K-Fold Configurations in Multi-Class Classification Problems

K-fold cross-validation is a widely used tool for assessing classifier performance. The reproducibility crisis [6] faced by artificial intelligence partly results from the irreproducibility of reported k-fold cross-validation-based performance scores. Recently, we introduced numerical techniques [10] to test the consistency of claimed performance scores and experimental setups. In a crucial use case, the method relies on the combinatorial enumeration of all k-fold configurations, for which we proposed an algorithm in the binary classification case. In this paper, the algorithm is generalized to multi-class classification.