On Chance Performance in High-Dimensional Class-Imbalance Problems

Generally, the area under the receiver operating characteristic curve (AUC) is considered to be a reliable performance measure in developing models where a class imbalance exists in the dataset. For such models, performance is adjudged on how much the AUC departs from a chance threshold of 0.5. However, the reliability of such threshold in class imbalanced problems is a growing concern. This paper investigates the severity of obtaining a high performance purely on chance by manipulating minority sample size, feature dimension, validation approach and classifier type.