A-A KD: Attention and Activation Knowledge Distillation

We propose a knowledge distillation method named attention and activation knowledge distillation (A-A KD) in this paper. By jointly taking advantage of the attention mechanism as an inter-channel method and activation information for intra-channel, the student model can overcome the insufficiency of feature extraction and effectively mimic features of the teacher model. A-A KD can outperform the state-of-the-art performance in various tasks such as image classification, object detection, and semantic segmentation. It can improve 1.8% of mAP on PASCAL VOC07 and 1.5% of mIoU on PASCAL VOC12 than conventional student models. Moreover, experimental results show that our student model (ResNet50) can reach 21.42% of top-1 error with A-A KD, which is better than the corresponding teacher model in ImageNet.