ASAP: A 28nm Transformer Training Accelerator with Alternating Sparsity and Asymmetrical Microscaling Floating-Point Precision

This paper presents a new Transformer training accelerator, ASAP, which supports alternating sparsity and asymmetrical precision, preserving accuracy while efficiently handling both dense and sparse weights. A reconfigurable pipeline SIMD (RP-SIMD) architecture is proposed to efficiently support special functions. ASAP achieves 26.6 TOPS/W energy efficiency and 2.9 TOPS performance, surpassing state-of-the-art works by 2.02-to-5.07× in training energy-efficiency.