A Study on improved vision transformer using Self-KD

Recently, the Vision Transformer (ViT) model has attracted attention for its outstanding performance in image classification. However, ViT requires a large amount of learning data and high computational resources, so it has low learning efficiency and is limited in real-time applications. In this study, we propose a method to simultaneously improve learning efficiency and generalization performance by applying the Self-Knowledge Distillation (Self-KD) technique to ViT to overcome these limitations. Self-KD can induce knowledge transfer effects without an additional teacher model by reusing the prediction information generated by the model during learning. In this study, we experimentally verified the effectiveness of the Self-KD technique using the CIFAR-10 dataset, and as a result, the ViT model with Self-KD applied showed about 3% accuracy improvement and faster convergence speed compared to the existing ViT. This means that ViT can be learned more efficiently even in limited computational environments, and suggests its potential for use in various real-world applications such as autonomous driving, medical image analysis, and face recognition. In the future, we plan to examine the generality and expandability of this technique through follow-up studies using large-scale datasets such as ImageNet.