Incremental Learning of Single-stage Detectors with Mining Memory Neurons

Object detection has made significant progress in recent years. However, there are still open questions regarding the incremental learning of object detectors. When we additionally detect objects of new classes on the trained model without the original training data, they suffer from “catastrophic forgetting” completely or abruptly forgetting previously learned classes upon learning new classes. Humans typically do not show catastrophic forgetting. Unlike prior work that uses knowledge distillation losses to maintain accuracy on original classes, this paper presents a simple and effective method for adding new class to the original deep object detectors while avoiding catastrophic forgetting. Our method is based on the framework of single-stage object detection. The core of our proposed solution is mining memory neurons in original model, then the rest neurons are updated and employed to detect new class. This strategy ensure minimal drop in performance of original classes. We conduct extensive experiments on the PASCAL VOC 2007 dataset, along with a detailed empirical analysis of our approach, and achieve much better performance against catastrophic forgetting than prior incremental object detectors.