Improving Performance of Vision Transformer on Skin Lesion Detection Leveraging Domain Associated Transfer Learning

To prevent lethal consequences, identification and early detection of different skin lesions are required. However, lots of expertise and expenditure are needed to detect them in the initial stages. Therefore, the requirement of an automated system that can detect and classify the types of skin lesions from image data is necessary. The proposed research is an attempt to design such an automated system which is based on a recently published dataset namely Hospital Italiano de Buenos Aires - Skin Lesions Images (2019–2022), consisting of 1616 images in total. This research proposes a noble approach that uses transfer learning and image augmentation to classify different types of skin lesions from image data in the most accurate way. Initially the base vision transformer model (ViT) of Google was used on the dataset and the resultant accuracy was low (0.70) because of imbalanced class distribution and lack of samples. Afterward, the domain-associated transfer learning approach was used through the HAM10000 Dataset by training the base ViT model with it and the model with the best validation accuracy was saved. In the next step, fine-tuning was done on the pretrained model by the actual dataset again and the validation accuracy improved by 4% from 70% to 74%, with increased recall values for some of the classes. Thus, the proposed approach showcases the usefulness of using domain-associated transfer learning for skin lesion detection.