What Can Attention Module Do in Knowledge Distillation?

Knowledge distillation is an effective method to transfer knowledge from teacher model or ensemble models to student. KD based on the intermediate layers makes the feature representations of student imitate the teacher. However, the feature maps of teacher contain less obvious information levels, which limits student’s efficient learning. In this paper, we propose a knowledge distillation method based on attention module representation (AM-KD). We believe that the feature representation of attention network with richer information levels can help student to learn the knowledge of teacher preferably. By multiplying the attention weight and the original network feature representation, we obtain knowledge with both hierarchical and global information. Experiments are carried out on CIFAR and Tiny-ImageNet datasets to verify the effectiveness of our method. When there is a large gap between the student and the teacher, the improvement of AM-KD is higher than that of the distillation method using attention weight alone.