Zero-Shot Fused Attention-Based GPT-2 Accelerator for Resource-Constrained Embedded Platform

This letter proposes a hardware-software co-design approach to accelerate inference of Generative Pre-trained Transformer (GPT-2) for resource-constrained embedded applications. Essentially, a standard configuration of GPT-2 (Python-based software implementation) is re-defined with high-level language (C++) to ultimately design a dedicated and optimized hardware logic of GPT-2 as an IP core, taking resources available on the ZCU104 FPGA board into account. The approach leverages a zero-shot learning setup, buffer tiling, and compiler directives for implementing a fused attention-based GPT-2 architecture with ZCU104, ensuring maximum computational power is effectively squeezed from available resources and trade-off between throughput (samples/second), power consumption (W), energy efficiency (mJ), and resource utilization is balanced. The proposed optimizations improve throughput by 25.3% (from 67.11 to 84.03 samples/sec) compared to the baseline. Moreover, a comprehensive investigation of the proposed optimization is done by leveraging the impact of layer fusion on latency, utilization, and throughput. Also, the generalizability of the proposed approach is validated by implementing various configurations of GPT-2. Codes and subsequent files are available at GitHub Repository.