An Improved Attention Transfer Teacher-Student Model for Edge Deployment

Large-scale deep learning models have done really well in things like understanding language, recognizing images, and other areas. However, these models also bring challenges, including computational complexity and storage requirements, which will induce the difficulty to implement large-scale deep learning models on edge computing devices. This paper proposes an improved attention transfer teacher-student model and related distillation loss function and training strategy, which is a teacher-student model knowledge distillation method for attention transfer through class activation maps. Not only does the teacher model change its predictions, but it also shares its attention with the student model, the goal is to make sure the student model works like the teacher model, so that the former can successfully learn the teacher model and be deployed at the edge of resource constraints device. The test results on open source datasets initially show that this method can further compress models and is suitable for deployment on resource-constrained devices.