Comparison of the potential between transformer and CNN in image classification

Convolution Neural Network (CNN) based algorithms have been dominating image classification tasks. In the meantime, Transformer based methods have also started to gain popularity and usage in recent years. To get a clear view and understanding of the two types of methods for image classification tasks on a butterfly dataset of 10,000 data points, the study is designed to compare the efficiency of CNN’s Insception-ResNetV2 model and Vision Transformer (ViT). For each of the methods, we also compare internally by different sizes of the dataset. By examining the experiment results of both validation accuracy and training time, we conclude that the ViT model’s accuracy is much more sensitive to largescale datasets. Also, ViT training requires relatively higher expense and duration. Meanwhile, the ViT model displays a relatively stable loss throughout the training process, illustrating feasible industry-level applications and opportunities for further refinement.