Loading [MathJax]/extensions/TeX/boldsymbol.js
SADIMM: Accelerating -parse -ttention Using --Based Near-Memory Processing | IEEE Journals & Magazine | IEEE Xplore

SADIMM: Accelerating \underline{\text{S}}S―parse \underline{\text{A}}A―ttention Using \underline{\text{DIMM}}DIMM―-Based Near-Memory Processing


Abstract:

Self-attention mechanism is the performance bottleneck of Transformer based language models. In response, researchers have proposed sparse attention to expedite Transform...Show More

Abstract:

Self-attention mechanism is the performance bottleneck of Transformer based language models. In response, researchers have proposed sparse attention to expedite Transformer execution. However, sparse attention involves massive random access, rendering it as a memory-intensive kernel. Memory-based architectures, such as near-memory processing (NMP), demonstrate notable performance enhancements in memory-intensive applications. Nonetheless, existing NMP-based sparse attention accelerators face suboptimal performance due to hardware and software challenges. On the hardware front, current solutions employ homogeneous logic integration, struggling to support the diverse operations in sparse attention. On the software side, token-based dataflow is commonly adopted, leading to load imbalance after the pruning of weakly connected tokens. To address these challenges, this paper introduces SADIMM, a hardware-software co-designed NMP-based sparse attention accelerator. In hardware, we propose a heterogeneous integration approach to efficiently support various operations within the attention mechanism. This involves employing different logic units for different operations, thereby improving hardware efficiency. In software, we implement a dimension-based dataflow, dividing input sequences by model dimensions. This approach achieves load balancing after the pruning of weakly connected tokens. Compared to NVIDIA RTX A6000 GPU, the experimental results on BERT, BART, and GPT-2 models demonstrate that SADIMM achieves 48\boldsymbol{\times}, 35\boldsymbol{\times}, 37\boldsymbol{\times} speedups and 194\boldsymbol{\times}, 202\boldsymbol{\times}, 191\boldsymbol{\times} energy efficiency improvement, respectively.
Published in: IEEE Transactions on Computers ( Volume: 74, Issue: 2, February 2025)
Page(s): 542 - 554
Date of Publication: 15 November 2024

ISSN Information:

Funding Agency:


Contact IEEE to Subscribe

References

References is not available for this document.