A Low-Complexity and Reconfigurable Design for Nonlinear Function Approximation in Transformers

Nonlinear function approximation plays a critical role in the area efficiency and accuracy of transformer-based hardware accelerators. Functions such as softmax, GELU, SiLU, and layer normalization require high numerical precision, introducing significant hardware complexity. Existing approaches reduce cost through functional integration but suffer from area overhead—particularly due to cubic preprocessing in GELU approximations. This paper proposes a novel low-complexity successive approximation technique that minimizes preprocessing complexity and compensates for errors across stages. A reconfigurable architecture is introduced to support softmax, GELU, SiLU, and square root operations within a unified hardware design. Evaluated on BERT-base models, the proposed solution achieves up to 2.00× and 3.99× improvements in area-normalized throughput on ASIC for softmax and GELU, respectively, and a 1.52× improvement in LUT-normalized throughput on FPGA, while maintaining model accuracy. For SiLU, our method reduces approximation error by 27.2% and improves LLaMA3.2-1B perplexity by 0.56% over the prior state-of-the-art design.