A Fully-Integrated Energy-Scalable Transformer Accelerator for Language Understanding on Edge Devices

Text and spoken language have become standard ways to interface with edge devices. Due to tight power and compute constraints, they require efficient natural language processing that can adapt the computation to the hardware conditions. This paper presents a Transformer accelerator running a custom-trained SuperTransformer model from which SubTransformers of various sizes can be sampled enabling adaptive model configuration. The test chip contains a variable-depth adder tree to support different model dimensions, a word elimination unit to prune redundant tokens, and dedicated softmax and layer normalization units. It achieves up to 6.9× scalability in network latency and energy between the largest and smallest SubTransformers, under the same operating conditions. Word elimination can reduce network energy by 16%, with a 14.5% drop in F1 score. At 0.68V and 80MHz, processing a 32-length input with our custom 2-layer Transformer model for intent detection and slot filling takes 0.61 ms and 1.6 μJ.