METAL: A Memory-Efficient Transformer Architecture for Long-Context Inference on FPGA

Transformer-based models have shown remarkable proficiency in extensive tasks for natural language processing, which are facing the ever-increasing need of processing long-context inputs. However, the memory footprint of the self-attention mechanism grows quadratically with the context length and becomes the bandwidth and memory bottleneck. Existing accelerators are mainly tailored for short sequences and struggle to handle attention in long-context scenarios. While some works attempt to mitigate this memory overhead with algorithmic optimizations, they suffer from limited hardware efficiency due to the sequential execution of backward-dependent iterations and additional computations. To this end, this paper proposes METAL, an algorithm-architecture co-optimized approach to support long-context inference with minimized memory overhead. First, we propose a hardware-friendly attention algorithm that eliminates the data dependency across inner loops, enabling full pipelining while keeping nonincreasing on-chip memory requirements, regardless of context length. Second, we develop a unified PE array for different dataflows across the transformer block to consistently support the entire inference with efficient data reuse. Moreover, an advanced non-linear operation module is designed to properly match the throughput of PE arrays with maximized resource sharing. Experimental results show that METAL on the Xilinx U200 FPGA outperforms other FPGA accelerators by $1.23-2.89 \times$ in normalized throughput and achieves up to $\mathbf{5 1. 0 \%}$ BRAM savings across different input sequence lengths.