Low-Cost Non-Linear Function Approximation for Transformer Deployment on Edge NPU

In recent years, Vision Transformers (ViTs) have outperformed Convolutional Neural Networks (CNNs) in accuracy on computer vision tasks. However, deploying ViT models on edge neural processing units (NPUs) presents significant challenges, particularly in quantizing the model for integer arithmetic and efficiently supporting non-linear functions. While several previous works have quantized ViTs and their non-linear functions, they rely on division operations, which are costly on edge NPU implementations. Additionally, these approaches are non-all-integer or require fine-tuning (i.e., quantization-aware training, QAT) to restore accuracy, which takes at least 60 GPU hours on V100. This paper proposes a low-cost, integer-only, division-free approximation that requires no fine-tuning (i.e., post-training quantization, PTQ) for non-linear functions. We apply integer-only Softmax (IntSoftmax), integer-only LayerNorm (IntLay-erNorm), and Critical-Range-Aware Quantization (CRAQ) for GELU. Our method introduces only a 0.5% accuracy drop compared to state-of-the-art non-all-integer PTQ approaches across the DeiT series. Furthermore, we demonstrate the feasibility of our algorithm on an NPU that supports both CNN and Transformer models. Compared to prior works in NPU hardware design for transformers, we reduce LUT consumption by 70% and eliminate additional DSP overhead. These results highlight our algorithm's low-cost, efficient nature, making it suitable for edge NPU deployment.