A Study on Efficiency Improvements of DNN Accelerators via Denormalized Arithmetic Elimination

Deep Neural Networks (DNNs) excel in tasks such as vision and speech processing, driving their widespread use. A key operation in DNNs is convolution, which is often implemented using General Matrix Multiplication (GEMM). Modern CPUs incorporate hardware accelerators for faster GEMM operation. Reduced precision floating-point (RPF) formats such as bfloat16 and TensorFloat32 are popular in machine learning, offering a balance between classification accuracy, dynamic range for training gradients, and hardware efficiency compared to traditional IEEE 754 32-bit or 64-bit floating-point numbers. While RPF is well-studied in the context of DNNs, a major complexity arises from handling denormalized numbers. This paper investigates their impact on DNNs and finds that removing denormalized number support from hardware accelerators does not affect classification accuracy, while allowing significant improvements in hardware efficiency and performance. With proliferation of data centers to accommodate the growing number of DNN applications, power reductions at the GEMM level can significantly improve cluster-wide efficiency, promoting sustainability. Our results indicate that eliminating denormalized arithmetic leads to significant improvements in multiplier circuit efficiency, with reductions of up to 46% in area (LUT count), 28% in longest path delay, and 59% in power consumption, all while maintaining DNN classification accuracy.