Structured Attention Knowledge Distillation for Lightweight Networks

Knowledge distillation is to transfer the effective knowledge learned by the teacher networks to the student networks through the designed loss function, which helps the student network to get better performance with less calculation cost. However, when there is a great difference between the student network and the teacher network in terms of structure and calculation amount, the previous knowledge distillation methods can hardly improve the student network performance effectively. In order to improve this situation, this paper proposes structured attention distillation for lightweight networks. Structured attention distillation groups the features from models by channels, which helps student networks learn the feature extraction ability of teacher networks by refining the spatial attention maps. Our proposed method is evaluated on CIFAR100 and large-scale face recognition validation set (LFW, CFP-FP, Age-DB). Compared with other distillation methods, our proposed method achieves better accuracy on CIFAR100 and face recognition validation set.