CORDIC Based Implementation of the Softmax Activation Function

Interest has been renewed in the domain of neuromorphic circuits, with the emergence of Deep Neural Networks as the leading model for various classification tasks. In this work, we present a CORDIC based implementation for one of the most important layers, the Softmax layer. The layer is particularly complex for CMOS implementations due to the requirement for multiple evaluations of the exponential function in calculating the output of this layer. We apply the CORDIC method for computing the exponent and the binary division algorithm for calculating the output. It is shown via experiments that our approach for realizing the Softmax function simultaneously out-performs existing implementations using McLaurin series approximations and Padé polynomial approximations from literature in three key metrics: timing, area, and accuracy. We also investigate an inverse form of the Softmax function that avoids the division operation. The implementation can be used to realize the Softmax activation in both ASIC and FPGA implementations of large scale neuromorphic circuits.