A Continuous Masked Method Filtered by Refine-Tuning Based on Encoder Architecture

Vision Transformers (ViT) have been shown to attain highly competitive performance for various computer vision tasks, such as image classification, object detection, and semantic segmentation. However, the task in which the ViT model is the most expert is still the image classification task. This paper also investigates the problem of optimizing ViT for the classification task. The ViT model achieves the highest accuracy on many image classification datasets by transferring the encoder part of transformer models. Following the initial success of the ViT model, there are still many problems ahead for ViT. These problems include the following three main points: difficulty in training models, lack of matched optimization methods, and difficulty in first filtering of designed optimization modules and optimization methods to save time. (1) To solve problems one and two: we have transferred a method called masked to the ViT model. Models using the masked method, such as Bert and MAE, have achieved good results. Similarly, our experimental results show that masked methods improve the training speed and the accuracy of the ViT model. (2) The MAE model uses a masked position selection method called the random masked. However, we find that ViT differs from these models of MAE and Bert in several ways, resulting in the random masked method, which worked well in MAE and Bert, performing less than expected in ViT. We fi that random masked optimization results in less than expected were due to a mismatch between the number of masked images and the number of labels after the random masked method. (3) To find an adapted masked position selection method for the ViT model, we experimented extensively with the fine-tuning method or retraining an initialized model. However, the fluctuations in the fine-tuning method were too small to be observed, and the cost of retraining an initialized model is unbearable. Therefore, we devised a refined-tune method to amplify the experimental results and used it to filter different masked position selection methods at first. (4) With the help of the fine-tuning method, we found continuous masked method is suitable for the ViT model. The number of masked images and labels for an original input image after continuous masked is in a suitable ratio. According to our experiments, dropping the ratio by another order of magnitude decreases the accuracy. Moreover, the number of masked images generated by continuous masked for an original input image is comparable to the number of training epochs.