Discriminative Gradient Adjustment with Coupled Knowledge Distillation for Class Incremental Learning

Class Incremental Learning (CIL) is a promising approach to addressing the catastrophic forgetting problem when learning for new categories. Though recent works based on dynamic architectures achieve convincing performance, data imbalance caused by limited size of memory and compression of the increasingly growing network are challenges to be solved. In this paper, we propose the novel Discriminative Gradient Adjustment (DGA) and Coupled Knowledge Distillation strategy (CKD) for these two challengs. The DGA mitigates the data imbalance problem by designing the loss function with a static global balance factor and a ground-truth-based dynamic factor. The CKD fully utilizes intermediate layers of the dual-branch models by feature-level distillation with moving-average weight updating for network compression. Extensive experiments on CIFAR100 and ImageNet100 datasets demonstrate the superiority of our method for CIL.