STPE: An Energy-Efficient Edge-Device Transformer Inference Processor with Multi-Mode Data-Compression Scheme

Transformer-Based models have turned out to be very successful in many artificial intelligence (AI) tasks, outperforming traditional convolutional neural networks (CNNs), especially in the field of Natural Language Processing (NLP). Their success relies upon a self-attention mechanism which, when compared to CNNs, has a global rather than a local receptive domain. This article proposes an energy-efficient edge-device Transformer inference processor termed Smart Transformer Processing Element (STPE). Firstly, STPE sets up a Multi-Mode Indexing and Sparsity Scheme (MISS) for token association, and further reduces the computational load through in-situ computation; secondly, STPE exploits the Local Properties of Attention Mechanism (LPAM) to further reduce redundant and repetitive calculations in Transformer operations by means of a search band calculation and error correction mechanism; thirdly, STPE has designed a Quantization and Compression Parallel Method (QCPM) to improve the computing speed and hardware utilization under weak related (WR) token. Employing 28nm CMOS synthesis tools, the area of the proposed STPE processor is 7.33 mm2. Its peak energy efficiency is 84.15TOPS/W, which is 14.7 times higher than that of the H100 graphics processing unit (GPU) and 3.06 times higher than that of the most advanced Transformer processor.