Refining Vision Transformer Models for Small-Scale Data with Locality Information from CNNs

Vision Transformer (ViT) architectures have become widely adopted in the field of computer vision. However, pure ViT models often underperform on small-scale datasets due to their reliance on large amounts of data for meaningful feature learning. To tackle this limitation, we propose an effective method to enhance ViT performance on small datasets by incorporating locality information from Convolutional Neural Networks (CNN). Given the strength of CNN in local feature extraction and ViT in global feature learning, we introduce a fusion framework that leverages pre-trained CNN features to guide ViT models. This integration combines CNN's local inductive bias with ViT's global receptive field, resulting in improved performance on small-scale datasets. Specifically, our approach enhances the locality awareness of ViT without compromising their global feature representation capabilities. We provide a detailed implementation of this fusion method and evaluate its effectiveness through experiments on the Tiny ImageNet dataset [3]. Our results demonstrate significant performance improvements across various ViT models, including an 11.30% improvement for ConViT, a 20.82% improvement for T2T, and a 19.62% improvement for PiT.