16.3 An on-Device Generative AI Focused Neural Processing Unit in 4nm Flagship Mobile SoC with Fan-Out Wafer-Level Package

A notable trend observed in on-device AI is natural progression from camera application-centric CNN-based neural networks to transformer-based generative AI (Gen AI) [1]. For instance, large language models (LLM) such as LLaMA [2] can support natural language understanding and human-like text generation while large visual models (LVM) such as Stable Diffusion [3] can generate images or 3D models based on user context. However, gen AI models exhibit different operational characteristics from traditional neural network (NN) models. LLMs require reading a several GB of weight data from DRAM every time a single token is generated during decoding, resulting in memory-intensive behavior. LVMs, on the other hand, are more compute-intensive than LLMs but have distinct operational characteristics, with softmax and layernorm accounting for 40% of total computation time [4], whereas CNNs typically consist of convolutions that account for the majority (90 to 99%) of operations in these networks. We report on neural processing unit (NPU) in 4nm Samsung Exynos™ 2400 that employs heterogeneous architecture consisting of vector engines and two types of tensor engines. NPU integrates a memory hierarchy and tiling techniques to support a wide range of neural networks. AI performance is boosted by enhancing heat dissipation through fan-out wafer level packaging (FOWLP).