Implementation of Activation Functions using various approximation methods

This paper compares the performance of activation function hardware under exponential function approximation techniques. The activation function is a key component of deep neural networks, allowing them to have nonlinear properties. Activation functions use exponential functions to obtain nonlinear properties, which make hardware implementation challenging. Exponential function hardware is implemented using representative approximation techniques such as Look-up-table, CORDIC, and Taylor series expansion. In addition, an approximation technique suitable for the activation function are suggested by performing an analysis of the hardware operation accuracy and area on the activation functions Sigmoid, Tanh, Swish, and GELU. Synthesis result shows CORDIC is area-efficient than other techniques in all functions. For accuracy, the Taylor series expansion in Tanh is the best, and CORDIC is good in the other activation functions.