An FPGA-Based Frequency-Focused Vision Transformer Accelerator for Real-Time Inference on Edge Platforms

Vision Transformers (ViTs) often struggle to effectively capture local features. SpectFormer addresses this limitation by incorporating spectral blocks into the shallow layers of DeiT. However, deploying SpectFormer on edge devices is challenging due to its high computational complexity, the implementation of spectral blocks, additional matrix transpositions, and intensive nonlinear operations. To address these challenges, we propose an FPGA-based Frequency-focused Vision Transformer Accelerator (FFVTA), the first FPGA-based ViT accelerator that incorporates frequency-domain optimization for real-time inference of SpectFormer on edge devices. FFVTA employs a Unified DFT-Attention Matrix (UDAM) architecture to unify the computation of spectral and self-attention blocks, significantly enhancing the hardware reuse and reducing resource usage. Additionally, FFVTA introduces a Block-Broadcast Loop (BBL) Dataflow to efficiently accelerate matrix multiplication in self-attention blocks and a Configurable Two-Stage Log-Softmax (CTS-Log-Softmax) computation process to optimize nonlinear function computations. These innovations improve computational efficiency while reducing the resource usage. When deployed on a resource-constrained FPGA platform (KV260), FFVTA achieves a 16.61× speedup compared to the baseline, with significant reductions in the hardware resource usage and power consumption. FFVTA demonstrates superior performance with an energy efficiency of 63.40 GOPs/W and a DSP efficiency of 0.46 GOPs/DSP. FFVTA not only achieves efficient acceleration for the hierarchical SpectFormer-H-S variant, but also attains competitive accuracy of 84.02% on ImageNet classification.