Stratified Cross-Validation on Multiple Columns

Stratified cross-validation is one of the standard methods of how to evaluate classifier’s generalization accuracy. However, conventional implementations of cross-validation can stratify only by a single column. In this paper, we propose to utilize Integer Linear Programming in order to enable stratification by multiple columns. Our experiments using an extensive set of multi-label data sets shows that the proposed method significantly outperforms non-stratified cross-validation.