Performance assessment of knowledge distillation models using the knowledge retention score

In this study, we introduce the Knowledge Retention Score (KRS) as a universal performance metric to evaluate knowledge distillation (KD) models. Unlike traditional evaluation methods that focus solely on final accuracy, KRS captures both intermediate feature alignment and output consistency between teacher and student models. This dual-component metric, combining feature similarity and output agreement, provides a comprehensive understanding of knowledge transfer effectiveness. We applied KRS across various KD techniques, teacher-student network pairs, and tasks, including image classification, object detection, and image segmentation, using multiple datasets such as CIFAR-100, COCO, and Oxford-IIIT Pet. The results revealed that feature-based KD methods achieved higher KRS in object detection tasks, indicating superior knowledge retention. In contrast, relational-based KD performed well in image segmentation but did not surpass feature-based KD. Response-based KD showed the lowest KRS, particularly in image classification tasks. The findings validate KRS as a robust and consistent metric, offering insights into the efficiency and effectiveness of different KD strategies across diver se contexts. This study sets the groundwork for future benchmarking and optimization of KD models using KRS, advancing the field of model compression and deployment on resource-constrained devices.