Adaptive Token Pruning for Vision Transformer

Aiming at the challenges of Vision Transformer (ViT) model in terms of computational cost, an adaptive pruning strategy for tokens based on Closeness-Centrality is innovatively proposed. The strategy cleverly combines the Euclidean distance between token features with the graph theory algorithm to realize the accurate grouping of tokens in the ViT model. Based on the grouping, we further evaluate the centrality of each group of tokens and selectively prune the tokens with lower centrality accordingly, thus significantly reducing the computational complexity while maintaining the excellent performance of the model. Compared with existing methods, this strategy fully considers the influence of location information on feature similarity, effectively improves the recognition accuracy of similar features, and reduces the feature loss introduced by merging. Experimental results show that in the ImageNet 1K classification task, our method not only achieves a good balance between accuracy and speed, but also successfully merges the various parts of the object in the image, demonstrating high accuracy and efficiency. In addition, the strategy achieves excellent performance on different kinds and sizes of models, and the balance between accuracy and speed can be achieved without additional training, which provides a more cost-effective solution for the application of ViT models.