An Efficient Layer Normalization Training Module With Dynamic Quantization for Transformers

Layer normalization (LN) function is widely adopted in Transformer-based neural networks. The efficient training of Transformers on personal devices is attracting attention for data privacy and latency concerns. However, the critical LN function involves extreme outliers for quantization, as well as hardware-unfriendly square-root and division operations, posing resource challenges for training deployment on the edge. This brief proposes an efficient LN training architecture with algorithm and hardware co-optimization. Specifically, we present a dynamic quantized algorithm based on integer arithmetics to smooth outliers for sufficient training accuracy. Then, we develop a reconfigurable hardware architecture to efficiently support various operations during LN training, with a vector-wise pipelined dataflow to improve hardware efficiency further. Experimental results show that our architecture achieves up to 0.25 and 1.0 Giga input per Second (GinS) in throughput at FPGA and ASIC platforms, respectively, outperforming prior works.