A Low-Cost Reconfigurable Architecture for Efficient Softmax and GELU in Transformers

Efficient computation of non-linear functions is essential for optimizing the area and performance of CNNs and DNNs. Among these functions, the softmax function poses challenges due to its normalization process, which involves summing exponential values after maximum extraction, increasing latency and limiting hardware utilization, particularly in re-source-constrained environments. While online normalization mitigates these issues with iterative updates, it introduces additional hardware overhead and lacks flexibility for supporting other non-linear functions. This paper presents a reconfigurable unit for building resource-efficient architectures, supporting both the softmax function with online normalization and GELU functions, achieving high performance in Transformer models with reduced hardware costs.