A Vision Transformer Approach for Classification an A Small-Sized Medical Image Dataset

In natural language processing, transformers have supplanted RNN models as more advanced technology. The vision transformer (ViT) model which is built upon the transformer architecture brought the concept of self-attention to images. Though the ViT model has given a better performance than CNNs on huge datasets, it still gives a poor performance on small-sized datasets when trained from scratch. This is because of its low locality inductive bias problem. This paper proposes a modified vision transformer architecture that would consist of three additional layers. A modified tokenization layer, a token learner layer, and a stochastic depth layer. The model is trained from scratch on a small-sized medical image dataset which is HAMS10000. The model is then compared with inception ResNet V2, a Vgg16 model, and a conventional ViT model trained on the same dataset. We see a more than 10% improvement in the ViT model and a significant increase in accuracy compared to the CNN models.