Similarity-Aware Fast Low-Rank Decomposition Framework for Vision Transformers

Vision transformers (ViTs) have shown success in computer vision tasks. However, their high computational and memory demands limit on-device implementation. Low-rank decomposition (LRD) is widely acknowledged as an effective method for reducing computational complexity. Nevertheless, previous approaches relying on heuristic-based or network architecture search (NAS) methods consume extensive searching and training time to uncover the optimal ranks. To efficiently discover rank distributions, this paper introduces a Similarity-Aware Fast-LRD framework, leveraging a greedy selection metric to optimize the compression ratio within each weight matrix by considering cosine similarity and rank reduction. Notably, our proposed framework restores accuracy in 15-25 epochs of model fine-tuning, surpassing the 30-300 epochs needed in prior work. Furthermore, our automated rank search process consumes fewer than 4 GPU hours. Experimental results show that our proposed Similarity-Aware Fast-LRD framework reduces FLOPs and associated parameters by 49.6% and 38.9% for DeiT-B and Swin-S, respectively, with merely 0.83% and 0.72% accuracy degradation on ImageNet.