Leveraging Knowledge Distillation in Vision Transformers for Binary Classification of Kidney Tumors from CT Radiography Images

Kidney tumors are commonly diagnosed through computed tomography (CT) imaging, making accurate classification into normal or Cyst categories essential for effective treatment planning. In this study, we propose kidney tumor classification using knowledge distillation between a Token vision transformer (teacher model) and a Vision transformer (student model). The teacher model employs a token-based learning mechanism to capture spatially important regions of the CT images, which are then distilled into the student model to enhance its performance while maintaining computational efficiency. Our method processes CT radiography images by partitioning them into non-overlapping patches that are fed into transformer encoder layers. The teacher model, equipped with a Token Learner module, extracts salient tokens, while the student model is trained using both true labels and softened teacher predictions through temperature-scaled softmax. This knowledge distillation framework is tested on a binary classification task to distinguish normal from cyst kidney tumors. The results indicate that the student model achieves comparable accuracy to the teacher model, despite having fewer parameters and faster inference times. Specifically, the student model achieves a peak accuracy of 95.96% and a recall of 95.56% at α =0.1 and temperature T =9. These findings show that knowledge distillation allows the student model to perform efficiently, delivering high classification accuracy with reduced computational resources. This approach presents a viable solution for real-time medical image classification in resource-constrained settings and demonstrates potential applicability to other medical imaging tasks.