Teacher-Free Knowledge Distillation Based on Non-Progressive Meta-Learned Simulated Annealing

Knowledge distillation (KD) is one of the effective methods in model compression. Recently, some teacher-free knowledge distillation (TFKD) methods have been proposed to overcome the problem that the teacher model is too huge in the traditional KD. In this paper, we proposed a TFKD framework based on a non-progressive method that the target distribution will change during training. We use simulated annealing (SA) to control the variety of free-form distribution. In addition, we also developed a meta-learned reinforcement learning (RL) method to control the energy calculation in SA. We conducted the experiments on CIFARIOO, CIFARIO, and SVHN with the VGG-8 model. The accuracy of the VGG-8 model can be improved to 72.88%, which is higher than the baseline model by 2.44%. Compared to another teacher-free method currently, the performance of our method is also the highest.