DATA: A Memory-Efficient Deformable Transformer Accelerator via Neural Architecture Search

Transformer is widely used in the field of artificial intelligence (AI) due to its excellent feature extraction capabilities. Its variant, the deformable transformer, is highly appreciated in autonomous driving and robotics since it employs a deformable attention mechanism to enhance feature extraction. However, due to its out-of-order memory access and data dependency, the deployment of the deformable transformer on mobile devices is much limited. To address these problems, a deformable attention transformer accelerator (DATA) is proposed in this work to speed up the processing by co-optimizing the algorithm and hardware. Specifically, we propose a memory-aware neural architecture search (NAS) method for deformable attention by constructing a continuous search space to automatically obtain a memory-efficient feature map slicing scheme. Based on the proposed slicing scheme, we design an efficient data flow to avoid the memory access conflict problem. In addition, a space-division multiplexing and time-division multiplexing hardware computing module is introduced to perform computations in the deformable attention layer, which greatly improves the utilization of hardware resources. Finally, the proposed accelerator is implemented on an Xilinx platform. In comparison, the proposed method achieves a maximum $2.42\times $ improvement in computational efficiency over prior arts, and the memory access requirement is reduced to 12.5% of the baseline.