Ras-D: Reverse Attention-Enhanced Self-Distillation

Self-distillation, as a knowledge distillation method that does not rely on a teacher network, enhances the overall performance of a neural network by enabling self-learning through its own knowledge. However, traditional self-distillation methods suffer from significant differences in the representational capacity between shallow and deep layers, as well as insufficient utilization of shallow-layer information. To address these issues, this paper proposes a Reverse Attention-enhanced Self-Distillation (RAS-D) strategy. Unlike conventional self-distillation, RAS-D reverses the supervisory direction, where the output of the shallow classifier supervises the knowledge learning of the deep classifier. Additionally, a Triplet Attention (TA) module is embedded to enhance the representational ability of the shallow classifier's input features. To reduce computational overhead, the feature map-based supervision is removed. Distillation experiments conducted on various benchmark image classification datasets using ResNet18 demonstrate the superiority of RAS-D. On CIFAR-100, RAS-D surpasses all other advanced self-distillation approaches, delivering a 0.36% accuracy gain compared to the next-best-performing method. Moreover, compared with the baseline model, RAS-D achieves an average accuracy gain of 2.15% on CIFAR-10, CIFAR-100, and Tiny-ImageNet.