Curvature-Based Piecewise Linear Approximation Method of GELU Activation Function in Neural Networks

Artificial neural networks (ANNs) rely significantly on activation functions for optimal performance. Traditional activation functions such as ReLU and Sigmoid are commonly used. However, they have computational and hardware efficiency constraints. This work presents a unique piecewise linear (PWL) approximation approach for the Gaussian Error Linear Unit (GELU) activation function that uses curvature values. We split the GELU function into pieces based on curvature values and then used the minimal squares approach to figure out each segment’s PWL approximation function equations. To determine the most appropriate function expression with a specified number of segments, we employ an optimization technique that minimizes both maximum and average absolute errors. This demonstrates that the PWL approximation of the GELU function accurately converges to the original function as the piecewise points’ number increases. The proposed method is intended to improve hardware efficiency while maintaining high accuracy. EExperimental findings indicate that the PWL approximation of GELU obtains $\mathbf{9 7. 9 6 \%}$ accuracy on the MNIST dataset. This method offers a promising alternative for improving the performance and efficiency of ANN activation algorithms in hardware applications.