ViTframe: Vision Transformer Acceleration via Informative Frame Selection for Video Recognition

Vision Transformer (ViT) has achieved state-of-the-art performance in the computer vision field, showcasing the remarkable potential to become a dominant model in the future. However, the self-attention mechanism within ViT presents significant challenges in terms of computational requirements and storage demands. This limitation becomes particularly pronounced in video recognition tasks, where the computational complexity escalates proportionally with the number of input frames. Current efforts to enhance ViT’s efficiency mainly concentrate on exploiting sparsity within individual frames, which neglects the temporal redundancy across frames, leading to an unsatisfactory solution for video tasks. Alternatively, in this paper, we propose a Vision Transformer acceleration framework called ViTframe, which aims to omit temporal redundancy in the video by dynamically selecting informative frames fed into ViT for fast video recognition. We first introduce an informative frame selection algorithm to pick out the most representative frames from the video for a quick ViT inference and a result compensation mechanism to compensate for the accuracy loss incurred by the informative frame selection. Moreover, we offer a customized architecture to efficiently implement the ViTframe algorithm. We implement ViTframe in a 28nm technology node. Extensive evaluations verify the effectiveness of our proposal on speedup, energy, and accuracy.