A Hybrid CNN-Transformer Architecture for Image Classification on Small Datasets

Image classification is widely used in remote sensing detection, medical diagnosis and other fields. By incorporating and improving various deep learning algorithms, research on image classification has made great progress in recent years. However, most of the existing image classification algorithms are targeted at large dataset situations. For some special cases, such as satellite remote sensing, disease effects, etc., it is often difficult to obtain sufficient data for model training. Therefore, it is necessary to develop an image classification architecture for small datasets. Considering that CNN is good at capturing local spatial patterns through convolutional layers, and the Transformer architecture based on the self-attention mechanism can model long-term dependencies and understand global features, we proposed a Hybrid CNN-Transformer Architecture for image classification in small datasets. We designed a special feature fusion module to discretize the feature maps of CNN and Transformer by softmax function at the decision layer and then fuse them. We used the Resnet10l network as the CNN part and the VIT-B-16 commonly used in image classification as the Transformer part, and designed 20 sets of experiments to determine the optimal fusion ratio of the CNN and Transformer networks, using Top-l Accuracy, Top-5 Accuracy, Mean Recall, Mean Precision, and Fl-Score for the evaluation of the results. The optimal fusion ratio of the two networks was finally determined to be 0.755:0.255. To evaluate the performance of the hybrid architecture, we selected the classic MobileNetv2, VGG13, VGG16, ResNet101, and VIT-B-16 networks for comparison and tested using the small dataset ImageNet1000(Mini). The results show that our model achieves 78.07%, 94.80%, 77.99%, 80.06% and 79.01% Top-1 Accuracy, Mean Recall, Mean Precision, and F1-Score under ImageNet1000(Mini), respectively, which are significantly better than several other networks.