ClearKD: Clear Knowledge Distillation for Medical Image Classification

In recent years, computer-aided diagnosis (CAD) systems employing convolutional neural networks (CNNs) have achieved remarkable performance in medical image classification tasks. Despite this, deploying CNN-based CAD systems on medical equipment presents challenges due to their enormous computational and storage resource requirements. In this case, knowledge distillation reduces the cost of deploying CNNs by guiding a lightweight student network to learn from a robust teacher network. However, medical images have higher inter-class similarity than natural images, which makes it difficult for the teacher network to provide clear and accurate classification knowledge to the student network, resulting in the performance degradation of the student network. To address this problem, we divide the teacher predictions into clear predictions, ambiguous predictions, and misclassified predictions to analyze the interference caused by the similarity of medical images on knowledge distillation and propose a novel knowledge distillation frame-work, termed ClearKD. By enhancing ambiguous predictions and misclassified predictions with clear predictions as a reference, our ClearKD consistently provides high-quality teacher classification knowledge to the student network, increasing the ability of the student network to distinguish medical images. The experimental results on the skin lesions classification datasets (ISIC2019) and the brain tumor dataset demonstrate that our ClearKD outperforms existing state-of-the-art knowledge distillation methods in medical image classification tasks.