HLC: A Hardware-friendly Quantization and Cache-based Accelerator for Transformer

Transformer-based language representations have demonstrated superior accuracy in various natural language processing (NLP) tasks. However, their deployment on terminal hardware is challenging due to the involvement of dense matrix operations and complex data flows. In this paper, we propose a heterogeneous multi-core low-power architecture design based on cache (HLC) for TIFP11: top-K half-precision integerized floating-point format. A quantization method is proposed to enable a hardware-friendly model compression technique, TIFP11. Furthermore, parallelized process elements (PE) and storage mechanisms for TIFP11 are carefully designed. Additionally, a cache architecture based on heterogeneous multi-core systems that significantly enhances the efficiency of transformer operations is described. We implement the transformer model using appropriate hardware scheduling. The IFP11 were deployed and verified the performance during storage and calculation. Experimental results demonstrate that the transformer deployed on VCK190 achieves a latency of 9.31ms with a batch size of 32, resulting in a 37.9× speedup compared to the CPU platform and a 1.94× speedup compared to the GPU platform.