Handling Imbalanced and Poorly Separated Data: a Multi-Stage Multi-Group Machine Learning Approach

Poorly separated data and imbalanced data present major challenges to classifiers which often result in lower accuracy and reliability in making predictions. In this paper, we introduce the multi-stage classification construct in which ‘difficult-to-classify’ observations are placed into a reserved judgment region for delayed future classification. Such a design is well-suited for poorly separated data that are difficult to classify without committing a high percentage of misclassification errors. The misclassification constraints within the classifier can be finetuned to allow management of imbalanced data, guiding minority entities into the reserved judgement region to avoid being misclassified into the majority groups. We wrap the classifier with a fast feature selection heuristic based on particle swarm optimization. An exact combinatorial branch-and-bound algorithm is also implemented to measure the quality of the heuristic solutions. We apply this multi-stage multi-group machine learning framework to two real-life medical problems: (a) multi-site treatment outcome prediction for best practice discovery in cardiovascular disease, and (b) uncovering patient characteristics that predict optimal response to intra-articular injections of hyaluronic acid for the treatment of knee osteoarthritis. Both problems involve poorly separated data and imbalanced groups in which traditional classifiers yield low prediction accuracy (47% - 66%). The multi-stage BB-PSO/DAMIP manages the poorly separated and imbalanced data well and returns interpretable results with 82% - 97% blind prediction accuracy.