A Heterogeneous FTJ-Based Computing-in-Memory Architecture for Vision Transformer Acceleration via Hardware and Algorithm Co-Design

Vision Transformers (ViT) have set new benchmarks in computer vision tasks but are often limited by their high computational and storage requirements. This work presents HEFA, a heterogeneous Computing-in-Memory architecture based on Ferroelectric Tunnel Junctions (FTJ), designed to accelerate ViT. By integrating an FTJ-based CIM Engine and a Digital Reconfigurable Engine with dynamic pruning, HEFA achieves $1.36 \times \sim 35.7 \times$ speedup and $1.96 \times \sim 34.1 \times$ energy efficiency improvements compared to GPU and state-of-the-art accelerators.