A Study on Improving Vision Transformer Classification via CNN-Based Object Extraction Preprocessing

This study proposes a lightweight preprocessing technique that combines DeepLabv3, a Convolutional Neural Network (CNN)-based object segmentation model, and Vision Transformer (ViT) to improve the image classification performance of ViT even on datasets with few samples. First, the object area of the input image is segmented at the pixel level using the pre-trained DeepLabv3. Subsequently, by connecting the object-masked image and the original image at the channel level and using this as input to the ViT, the model is encouraged to explicitly learn object-oriented local cues while maintaining the global context. The segmentation network is fixed, and we only fine-tune the classification head of ViT, so the computational burden is only slightly increased. As a result of performing a 5k/1k subsample experiment on four benchmarks, the proposed pipeline achieved an accuracy gain of up to 9 percentage points (from 89% to 98%) and faster convergence. In particular, consistent performance improvement was confirmed even on SVHN with complex backgrounds and Oxford-IIIT Pet, which requires detailed class distinction. Self-attention analysis visually demonstrates that the ViT focuses on actual object areas. This method can be applied without additional learning steps or model structure changes, alleviating the ViT overfitting problem in small data environments and improving model interpretability. Therefore, it is expected to contribute to the establishment of a data-efficient and interpretable classification model in various computer vision applications such as autonomous driving and medical imaging.