An Energy-Efficient FPGA-Based Vision Transformer Accelerator via Software-Hardware Co-Design

The ViT models have a large number of parameters and intensive matrix computations, making it challenging to deploy on resource-constrained FPGAs for acceleration. In this paper, we propose an end-to-end quantized ViT accelerator that adopts a multi-kernel architecture and a time-multiplexed scheduling strategy. We implement customized hardware for key operations in ViTs, enabling efficient resource utilization and memory-friendly features. Experiments show that compared to Edge-MoE, our accelerator achieves an 8.64× improvement in energy efficiency. Meanwhile, hardware resource consumption is significantly reduced, making it more suitable for resource-constrained FPGAs.