Class-Imbalanced Deep Learning via a Class-Balanced Ensemble

Class imbalance is a prevalent phenomenon in various real-world applications and it presents significant challenges to model learning, including deep learning. In this work, we embed ensemble learning into the deep convolutional neural networks (CNNs) to tackle the class-imbalanced learning problem. An ensemble of auxiliary classifiers branching out from various hidden layers of a CNN is trained together with the CNN in an end-to-end manner. To that end, we designed a new loss function that can rectify the bias toward the majority classes by forcing the CNN’s hidden layers and its associated auxiliary classifiers to focus on the samples that have been misclassified by previous layers, thus enabling subsequent layers to develop diverse behavior and fix the errors of previous layers in a batch-wise manner. A unique feature of the new method is that the ensemble of auxiliary classifiers can work together with the main CNN to form a more powerful combined classifier, or can be removed after finished training the CNN and thus only acting the role of assisting class imbalance learning of the CNN to enhance the neural network’s capability in dealing with class-imbalanced data. Comprehensive experiments are conducted on four benchmark data sets of increasing complexity (CIFAR-10, CIFAR-100, iNaturalist, and CelebA) and the results demonstrate significant performance improvements over the state-of-the-art deep imbalance learning methods.

I. Introduction

Over the past decade, convolutional neural networks (CNNs) have gained remarkable popularity as they have significantly advanced the state-of-the-art in various real-world tasks, such as image classification [1], object tracking [2], and segmentation [3], [4]. The increased availability of labeled data, improvements in modern graphical processing units (GPUs), and the algorithmic breakthroughs in software have provided CNNs with great learning capacity to explore the hypothesis space that have never been successfully explored by the so-called “shallow” models. However, in various real-world applications, the training data often exhibit significantly imbalanced class distributions [5]–[8]. This is a classic difficult problem in traditional machine learning. Despite their great learning capacity, this problem has not gone away for CNN-based methods and their performances can still suffer when the data have a skewed distribution [9], [10]. Despite its fundamental importance, class imbalance in the context of deep representation learning has been under-researched [7], [8].