Evaluating Knowledge Transfer in the Neural Network for Medical Images

The performance of deep learning models, such as convolutional neural networks (CNN)s, is highly dependent on the size of the training dataset. Consequently, it can be challenging to achieve satisfactory performance when training models from scratch in low-data environments. To address this issue, using knowledge transfer approaches from pre-trained networks can be particularly useful. In this study, we implement different experiments for standard transfer learning approaches as our baseline and introduce a novel knowledge transfer approach, called teacher-student learning, to improve the performance of predictive models in diagnostic medical imaging. Specifically, we investigate various configurations in the teacher-student learning framework inspired by the activation attention transfer in computer vision models to help address some challenges faced in medical imaging, such as the limited availability of annotated data and limited computing resources. We show that the teacher-student learning approach holds great promise in significantly enhancing the performance of diagnostic models. The implications of our findings could be instrumental in improving healthcare accessibility and affordability as they may enable the development of cost-effective and widely accessible medical imaging technologies, particularly in limited data environments.