Multi Step Knowledge Distillation Framework For Action Recognition in Still Images

Human Action Recognition (HAR) is a challenging computer vision task due to issues like lack of temporal information, significant intra-class variations, cluttered backgrounds, and misleading objects. A straightforward solution to cope with these challenges is to fine-tune complex deep neural networks that have been properly pre-trained on large scale datasets, such as Image-Net. Nevertheless, these models are computationally inefficient, making them unsuitable for real-world applications. One of the most popular approaches to this problem is knowledge distillation. It involves transferring knowledge from large network to computationally efficient model. Typically, these two networks are referred to as student and teacher networks. Furthermore, several studies have shown that despite stronger models providing more robust guiding signals for the training of student model, the substantial computational gap between student and teacher networks can negatively impact knowledge transfer. To deal with these issues, we propose an innovative multi step knowledge distillation framework that takes advantage of a mid-size teacher assistant network to narrow the computational gap between student and teacher networks. Also, self-distillation is used to improve teacher network performance, which in turn increases student network accuracy. Three variants of EfficientNetV2 are used to construct teacher, teacher assistant, and student networks. According to the results, based on Mean Average Precision (MAP) the student model performance on the Standford-40 dataset improved from 94.75% to 96.30% by employing the proposed knowledge distillation framework, which is better than the results reported by previous studies.