A 28nm 64.5TOPS/W Sparse Transformer Accelerator with Partial Product-based Speculation and Sparsity-Adaptive Computation

Transformer-based models have achieved notable success across various fields, thanks to the Multi-Head Attention (MHA) mechanism. However, their high computational and memory demands pose challenges for deployment on edge devices. This paper introduces a sparse Transformer accelerator with an algorithm-hardware co-design approach. Atthe algorithm level, sparsity-aware training algorithms are used to exploit both dynamic and static sparsity, reducing computation in Transformer models by $\mathbf{9 2\%}$. At the hardware level, the accelerator includes three key features to improve efficiency: 1) the Partial Product-based Attention Speculation Unit (PASU) dynamically speculates and skips unimportant computation, 2) the Efficient Softmax Engine (ESE) leverages redundancy in non-uniform data distribution, and 3) the Sparsity-Adaptive Computation Engine (SACE) dynamically dispatches sparse data to different computing cores, enhancing throughput. Fabricated using TSMC 28nm CMOS technology, the chip achieves a peak energy efficiency of 64.5 TOPS/W under 92% sparsity. Additionally, it optimizes the heterogeneous operation of the Softmax function, achieving $\mathbf{1 7. 9 3}$ TOPS/W for the entire MHA block.