23.8 An 88.36TOPS/W Bit-Level-Weight-Compressed Large-Language-Model Accelerator with Cluster-Aligned INT-FP-GEMM and Bi-Dimensional Workflow Reformulation

Large language models (LLMs) have shown remarkable performance across a wide range of natural language processing (NLP) tasks, becoming an essential part of modern society [1]–[4]. This exceptional performance can be attributed to huge model size and autoregressive computation [5], [6]. However, these attributes pose challenges for the efficient deployment of LLMs from 3 aspects, as shown in Fig. 23.8.1. First, an LLM has enormous external memory access (EMA) due to its autoregressive feature. During inference, it generates output tokens one by one until all the outputs are complete. In each iteration, the processor reads all model weights and the key-value (KV) cache for computation, while also storing a newly generated KV into the cache for further use. Hence, an LLM requires over 1000x EMA compared to a non-autoregressive Transformer [7], [8]. Second, an LLM may use a unique integer-float (INT-FP) mixed-precision general matrix multiplication (MP-GEMM). Due to accuracy concerns, weights can be quantized to INT, but the activations should be retained in FP [9]–[11]. Generally, processors dequantize INT weights to FP, turning MP-GEMM to FP-GEMM. This approach does not fully exploit quantization benefits and consumes 4.6x more power. Third, apart from GEMM, LLMs frequently use non-linear functions, such as trigonometric functions and softmax, which are commonly computed by a special function unit (SFU). The GEMM, as performed by a processing element (PE), depends on the non-linear results and thus waits for the SFU. Current processors fail to provide sufficient SFU throughput, resulting in PEs left waiting for 1/3 of the time [12], [13]. These problems hinder efficient LLM deployment for both cloud and edge devices, and are largely unexplored by existing accelerators which focus on traditional Transformers [13]–[22] or are limited to specific algorithms [23], [24].