CPSAA: Accelerating Sparse Attention using Crossbar-based Processing-In-Memory Architecture

The attention mechanism requires huge computational efforts to process unnecessary calculations, significantly limiting the system's performance. Researchers propose sparse attention to convert some DDMM operations to SDDMM and SpMM operations. However, current sparse attention solutions introduce massive off-chip random memory access. We propose CPSAA, a novel crossbar-based PIM-featured sparse attention accelerator. First, we present a novel attention calculation mode. Second, we design a novel PIM-based sparsity pruning architecture. Finally, we present novel crossbar-based methods. Experimental results show that CPSAA has an average of 89.6X, 32.2X, 17.8X, 3.39X, and 3.84X performance improvement and 755.6X, 55.3X, 21.3X, 5.7X, and 4.9X energy-saving when compare with GPU, FPGA, SANGER, ReBERT, and ReTransformer.


I. INTRODUCTION
Attention-based neural network shows accuracy leaps in machine learning applications, e. g., natural language processing (NLP) [10] and computer vision [8].Different from the commonly used Convolutional Neural Network (CNN) or Recurrent Neural Network (RNN) models, Transformer [30] adopts a pure attention-based neural network to better identify the dependencies between tokens of the input sequence.Following this design, Transformer and its variants achieve great accuracy improvement in NLP tasks [10], such as machine translation [30] and question answering [6], etc. Attention is also widely used in computer vision tasks [8] including image classification [1] and object detection [18], etc.
The vanilla attention mechanism [30] is usually implemented as DDMM and softmax operations.By computing an attention score matrix, the attention mechanism can pay attention to these relevant token pairs.There is overwhelming computation pressure in processing these irrelevant token pairs, leading to intolerable execution time [7].Researchers propose sparse attention by adding a sparsity pruning phase before the attention calculation to reduce irrelevant calculations [7], [19], since most tokens in the input sequence are unrelated to the current query.There are two types of sparse attention designs, i.e., software-based and software-hardware co-design methods [19].Software-based methods [29], [38] aim to propose various optimization algorithms to reduce computational overhead by increasing sparsity.Software-hardware co-design solutions accelerate sparse attention by taking advantage of high-parallelism hardware, such as Field Programmable Gate Array (FPGA) [16], [40] and Application Specific Integrated Circuit (ASIC) [7], [19], [32].
The above solutions only achieve limited speedups since both the sparsity pruning and attention calculation phases involve many off-chip data transfers.Emerging crossbar-based architectures are promising to solve the off-chip data transmission problem, such as Resistive Random Access Memory (ReRAM) and ReRAM-based content addressable memory (ReCAM) [12].ReCAM is suitable for high parallel comparison, the core operation of content-based similarity search in the attention mechanism.ReRAM is ideal for vectormatrix multiplication (VMM) operation, which has superior performance handling the DDMM operations of attentionbased neural network.Utilizing the in-situ processing ability of ReRAM arrays, there emerge ReRAM-based PIM-featured solutions to accelerate traditional neural network [27] and the dense attention mechanism [11], [36].However, these PIMbased solutions can hardly extend to accelerate the sparse attention for the following reasons.
First, the ReRAM array's write overhead cannot be ignored as it is in ReRAM-based CNN and RNN accelerators.Solving the ReRAM write overhead is urgent because many matrices in the attention mechanism cannot be reused and need to be written in runtime.Second, the sparse attention involves the sparsity pruning phase, which is not considered by current dense attention accelerators.Using the current software-based pruning algorithm can promote the PIM-based attention accelerator.However, the software-based pruning methods have poor performance because they need to load all input matrices from the off-chip memory to the processor.Moreover, the sparsity of the attention mechanism is pretty unstructured, which will introduce lots of off-chip random memory access to the attention calculation phase.Finally, the sparse attention involves the SDDMM and SpMM operations.Direct application of current ReRAM-based sparse methods [17], [28] to ReRAM-based SDDMM and SpMM operations will achieve inferior performance (for details to see § IV-C and § IV-D).
Given this landscape, we propose CPSAA, a novel Crossbarbased PIM-featured Sparse Attention Accelerator.First, we design the attention calculation mode to increase the parallelism of CPSAA dataflow while hiding ReRAM writing overhead.Second, we design a novel PIM-based sparsity pruning architecture to support our calculation mode while removing the off-chip data transmissions.Finally, we couple ReRAM and ReCAM arrays and propose new PIM-based SDDMM and SpMM methods to utilize the unstructured sparsity.The main contributions of this paper are as follows: • We propose CPSAA, a PIM-based sparse attention accelerator, to speed up both the pruning and attention calculation phases of the neural network inference.• We design the pruning phase as a novel PIM architecture to eliminate off-chip data transfers.The attention calculation architecture involves novel ReRAM-based SDDMM and SpMM methods to increase the parallelism of sparse attention.
• We evaluate and compare CPSAA with state-of-the-art attention mechanism solutions.The experimental results show that CPSAA has performance improvement in speedups and energy-saving.

II. BACKGROUND AND MOTIVATION A. Sparse Attention
The vanilla attention mechanism maps a query matrix Q and a key-value matrix pair K-V to an output matrix Z. Figure 1 (a) depicts the calculation procedure of the dense attention mechanism.First, the query (Q), key (K), and value (V ) matrices are obtained by multiplying the embedded input sequence X with the corresponding weight matrices, W Q , W K , and W V .Next, the score matrix S is calculated by multiplying Q with K T .Then, the S matrix is normalized with a row-wise softmax function Softmax(S).Finally, S is multiplied by the V matrix to get the output Z.
As Figure 1 (a) shows, S reveals the relevance between the tokens of Q and K matrices.Researchers find that most tokens in Q are irrelevant to K, making the S matrix inherently sparse [7].The above fact makes it possible to utilize the sparsity of S to avoid computing these irrelevant token pairs [7].Thus, people propose sparse attention, which adopts a sparsity ... pruning phase to save computational resources of the attention calculation phase.
SANGER [19] proposes a prediction-based pruning method and achieves high sparsity.The idea of their pruning method is to calculate the approximate score matrix S using a lowprecision DDMM operation, S = So f tmax(QU , where d is the dimension of the input embeddings, and QU(x) = round(γx) is a quantization operator that maps input buffers to low-bit by a scaling factor γ and a rounding bit-shift.QU −1 (•) is the corresponding de-quant operator that transforms low-bit outputs back into high precision.Then, a binarization procedure described in equation (1) will convert the S matrix to a binary mask matrix G, where θ is the threshold of this binarization function.Because the sparsity of the mask matrix is similar to the score matrix S, SANGER can convert the DDMM operation S = Q • K T to SDDMM operation.As Figure 1 (b) shows, SDDMM operation generates the sparse score matrix S with three input matrices (two dense matrices Q, K, and one sparse mask matrix), which can utilize the mask matrix to avoid calculating the zero-value (white cells) of the S matrix.Utilizing the sparse S matrix, people can convert the DDMM operation Z = S •V to the SpMM operation.Figure 1 (b) shows the visualization of the SpMM operation, which is a VMM operation between a sparse matrix S and a dense matrix V .

B. ReRAM and ReCAM Basics
Figure 2 (a) shows one ReRAM cell, which is comprised of a top electrode, a metal oxide layer, and a bottom electrode.Following the Kirchhoff's Current Law, ReRAM crossbar array can process VMM operation efficiently [33], denoted as shows.Utilizing this high parallel in-situ VMM operation, several ReRAMbased neural network accelerators [27] and graph processing accelerators [22], [28], [42] have been proposed.
The ReCAM array architecture is shown in Figure 2 (c), which is comprised of lots of two transistors and two memristors (2T2R) ReCAM bit-cells [12].One ReCAM bitcell contains a couple of ReRAM cells.One ReCAM array contains a Key register, a ReCAM cells array, drivers (DRV), and tag registers (TAG).ReCAM arrays can perform vectorscalar comparisons in parallel [12].The TAG will latch the '1' signal if one row matches with the Key register.

C. Related Work
Table I lists the recent studies of the attention mechanism.Different from SANGER's classification, we divide these works into three categories, i.e., software-based, softwarehardware co-design, and the new adding hardware-based.The software-based methods can be further divided into static and dynamic sparsity patterns.The static sparsity pattern can only get coarse-grained sparsity for data dependent [2], [38].The dynamic sparsity pattern can achieve higher sparsity conditioned on individual input samples during runtime, but their unstructured sparsity increases the random memory access overhead [3], [41].
Software-hardware co-design methods adopt more efficient hardware, such as FPGAs and ASICs.FTRANS [16] is an efficient acceleration framework for transformer-based NLP.Zhang et al. propose a novel pruning method and design the FPGA-based sparse attention accelerator [40].A 3 [7] can accelerate the attention mechanism with algorithmic approximation and hardware specialization.SpAtten [32] leverages token sparsity, head sparsity, and quantization opportunities to reduce redundent computation and random access.SANGER [19] presents a prediction-based pruning algorithm to reduce unnecessary calculations while designing the "splitting and packing" algorithm to reduce the massive random memory access overhead.DOTA [23] proposes a dynamic weak connections detector to avoid unnecessary attention calculation and promote sparse attention.
Although these co-design solutions can relieve the random memory access by designing hardware-specific algorithms, many off-chip data transmissions exist because of their separate memory and processor architecture.Using SANGER as an example, the "splitting and packing" algorithm can reduce random memory access by converting the unstructured sparsity to fine-grained structured sparsity.However, the benefit of the fine-grained structured sparsity comes at the cost of the dynamically configured control signals, which are highly complex to scheduling processing elements and memory access in runtime.In addition, SANGER introduces off-chip random access and data dependency to their sparsity pruning phase.Thus, no current accelerators can solve the off-chip random memory access problem elegantly.
We also list current hardware-based PIM-featured dense   I.These solutions use high parallel ReRAM arrays to significantly reduce the latency of DDMM operations [11], [36].However, since the sparsity of the attention matrices is not considered, current PIMbased attention accelerators need to calculate all five DDMM operation steps, wasting lots of computational resources on irrelevant tokens.Therefore, how to design a PIM-based sparse attention accelerator and reducing unnecessary calculations become the key to achieve further accelerations.

D. Motivation
We find that current sparse attention accelerators have many off-chip data transmissions.We design experiments on two sparse attention accelerators, i.e., SANGER and DOTA, to confirm this statement.Figure 3 presents the operations ratio of SANGER [19] and DOTA [23] running five real-world datasets (details to see Section V).The ratio of the operation comes from breaking down the response time to the mask matrix generation (MA-GE) and the attention mechanism calculation (AT-CA).To assess the impact of off-chip memory access, we further break down the MA-GE and AT-CA to the processor execution time (MA-GE-P and AT-CA-P) and the memory access time (MA-GE-M and AT-CA-M).The memory access time is obtained by subtracting the average kernel runtime from the total execution time.
First, the overhead of the mask generation and attention calculation phases both cannot be ignored.Figure 3 shows the MA-GE takes an average of 17.9% (14.3% in DOTA) response time, while the AT-CA takes 82.1% (85.7% in DOTA).The calculation overhead of the pruning phase is Second, most of the overhead of the mask generation phase comes from the off-chip memory access.The MA-GE-M takes an average of 94.6% (92.7% in DOTA) response time while the MA-GE-P takes 5.4% (7.3% in DOTA).The reason is that the MA-GE of SANGER (DOTA) needs to access Q and K from the off-chip memory, which involves many off-chip data transfers and takes lots of response time.
Third, the attention calculation also spends a lot of time on off-chip data transfers.The AT-CA-M takes an average of 71.2% (63.5% in DOTA) response time while the AT-CA-P takes 28.8% (36.5% in DOTA).That is because all the input matrices, Q, K, V , and S, need to be sent from the off-chip memory to the ASIC-based processor.These off-chip data transmissions greatly increase access latency and further hurts performance.The massive off-chip data transmissions motivate us to design a novel sparse attention accelerator to simultaneously accelerate MA-GE and AT-CA with pretty low off-chip random access.

III. CALCULATION MODE
In conventional ReRAM-based CNN and RNN accelerators, the write overhead of their weight matrices can be ignored because these matrices can be reused for different inputs.Unfortunately, the pretty high ReRAM write overhead of the K and V matrices in the attention mechanism cannot be ignored because these matrices vary for different input sequences.Even worse, the write operation works serial with the VMM operation and greatly increases the latency, just as the calculation mode of ReBERT [11] shown in Figure 4 (a).ReBERT uses the same calculation mode as SANGER, which has pretty high VMM parallelism since it can get Q, K, and V simultaneously.However, the VMM operation S = Q • K T has to wait for the write operations of matrices K T , which greatly hurts the performance of ReBERT.
To reduce the performance impact of ReRAM writing, Re-Transformer [36] depicts a novel calculation mode as Figure 4 (b) shows.To avoid the VMM operations waiting a long time for the writing of K T , the authors avoid generating K and V .With the weight matrices W Q , W K , and W V used in serial, their new computational mode achieves the goal that the VMM operations can work concurrently with the writing.However, three challenges limit this calculation mode to efficiently support sparse attention: 1) Data dependency.As Figure 4 (b) shows, ReTransformer has to sequentially get matrix Q → R → S → P → Z with high data dependency, while Q, K, and V matrices could be parallel computed in ReBERT.
2) Poor performance when extending to mask generation.Things get worse in sparse attention because there is a mask generation phase.Suppose ReTransformer wants to get the quantized S (i.e., mask matrix).ReTransformer should get Q first; then use the quantized Q to get the quantized R; finally, to get quantized S. Therefore, the generation of Q (DDMM operation) becomes the critical path of the mask generation.3) Data dependency between the mask generation and attention calculation.The mask generation of ReTransformer must wait for its attention calculation (i.e., data dependency).Because ReTransformer must generate Q and R matrices to get the mask matrix.Therefore, the mask generation of ReTransformer can not work concurrently with its attention calculation, i.e., ReTransformer does not see the potential of parallel execution between mask generation and attention calculation.
We develop a novel calculation mode to solve the above challenges, as Figure 4 (c) shows.We find that the above three challenges are the victims of the data dependency of Q.To unbind the data dependency of Q, we pre-calculate the weight matrix W S by performing W S = W Q • W T K , and we pre-store W S and W V in ReRAM arrays.The calculation of S described in equation ( 2) can be converted to equation (3) due to the combination law of matrix multiplication.Based on equation (3), we can get the score matrix S by performing two VMM operations, i.e., M = X • W S and S = M • X T .To further unbind the data dependency between P and Z in Figure 4 (b), we calculate V in advance as Figure 4 (c) shows.After V = X •W V is calculated, we can get the output Z matrix by performing S •V .Our novel computational mode can unbind the data dependency of Q and achieve parallel VMM operations, while the VMM operations can also run parallel with write operations in an asynchronous manner.Further, this new calculation mode can also significantly improve the performance of mask generation (for details to see Figure 7).The tiles assigned to the same attention head communicate with each other through on-chip bus.Because real-world applications usually need multiple attention layers working together, we also show the input matrix from the previous layer and the output matrix to the next layer, which is the off-chip data transmission managed by the data transfer controller (DTC A ).We use the circled color numbers to show the CPSAA dataflow, which is corresponded to the numbers in Figure 7.The functions of all components are as follows:

IV. CPSAA A. Overview
CTRL.The controller will generate control signals for all components in one Tile.
Buffers.The Input Buffer (IB) will store the input embeddings, i.e., X matrices.The Crossbar Buffer (CB) will store the matrices generated at running time.The address information table (AIT) will record the address information of matrices in the ReRAM arrays, which will be useful when we need to know the location of a specific sub-matrix.
QU and DQU.The Quant Unit will quantize the matrix to a low-precision representation (via QU(•) function), and the Dequant Unit will convert the low-precision representation back to high precision (via QU −1 (•) function).The architecture of the QU is shown in Figure 6 (a), and the DQU is designed in the inverse process.
SU and BU.The Softmax Unit will perform the softmax function, and the detailed architecture of the SU is the same as A 3 [7], shown in Figure 6 (b).The input of SU is matrix S, which is calculated by multiple crossbar arrays in one tile.While crossbar arrays calculate matrix S in different tiles, CPSAA will gather these results to generate S. The Binarization Unit will convert a given matrix to the '01' matrix, which is performed by a binarization comparator.
ReCAM Scheduler.The ReCAM arrays will store the mask matrices, which will be used to generate control signals for the ReRAM-based VMM operations.
ROA.The read-only arrays will pre-store those matrices that can be reused for different inputs, i.e., QU(W S ), W S , and W V .The ROA contains several Arrays Group (AG), and we set a read-only mark for the ROA.
WEA.The write-enable arrays will store these matrices that are generated in the runtime, i.e., QU(X T ), X T , and V .
AG.The Arrays Group ( C ) is the basic storage and computing units, which contains one shift and add Unit (S+A), one input register (IR), one output register (OR), one analogdigital-converter (ADC), and several ReRAM arrays.

B. Dataflow
The color-circled numbers in Figure 7 show two CPSAA visualization dataflows, i.e., the pruning phase's dataflow (bluecircled numbers) and attention calculation dataflow (yellowcircled numbers).To illustrate the dataflow more clearly, we divide the CPSAA dataflows into four steps, as Figure 7 shows.Step1 is a PIM-based pruning method to generate the mask matrix.Step2 contains the calculation of M = X • W S , V = X • W V , and the write of the matrix X T (Noting that Step1 and Step2 can run in parallel since we remove the datapath dependency of Q between these two steps).Step3 contains the SDDMM operation S = M • X T and the write of matrix V .Step4 performs the SpMM operation Z = S • V .It is worth noting that Step2-Step4 is the detailed version of the computational mode in Figure 4 (c).
Pruning Dataflow.The pruning algorithm of SANGER needs to use the full-precision Q and K matrices to generate the mask matrix, which makes the pruning phase wait for the calculation of these intermediate matrices Q and K. To avoid the waiting of the pruning phase (i.e., Step1 waiting for Step2), we fine-tune the mask generating algorithm as equation ( 4), where Bi(•) is the binarization function and So f t(•) is the softmax function, and the QU(X), QU(W S ), and QU(X T ) denotes the low-precision X, W S , and X T .Our new pruning algorithm can use the input embedding matrix and the weight matrix to calculate the mask matrix, which does not need to use the results of Step2, i.e., M or V .
As Step1 in Figure 7 shows, we pre-store the low-precision QU(W S ) in ROA.When an input matrix X arrives at CPSAA, the system will send X to the QU and get QU(X) ( 1 ).Then, QU(X T ) will be written to WEA ( 2' ), and a VMM operation will be performed between QU(X) and QU(W S ) to get QU(M) ( 2 ).Next, the system will perform a VMM operation between QU(M) and QU(X T ) to generate QU(S) ( 3 ).After that, QU(S) will be sent to the DQU, SU, and BU to get the mask matrix ( 4 ).Finally, the mask matrix will be written to a ReCAM array ( 5 ).The sparsity of the mask is similar to the matrix S, so we can use the mask matrix to convert the DDMM operation S = M • X T to the SDDMM operation.
Attention Calculation Dataflow.As shown in Figure 7, the weight matrices W S and W V are pre-stored in ROA.At the beginning, CPSAA receives and stores an input embedding matrix X to the Input Buffer ( 1 ).Then, the following three phases will be performed in parallel: 1) CPSAA will write X T to WEA ( 3' ), 2) CPSAA will send X to the IR via AIT to perform the DDMM operation M = X • W S ( 2 3 ), and 3) CPSAA will perform the DDMM operation V = X • W V to generate matrix V ( 2 3 ).After that, CPSAA can use mask matrix to perform the SDDMM operation S = M •X T to get the score matrix S while writing V to WEA ( 4 ).After CPSAA performs the softmax function on S ( 5 ), CPSAA will use the mask matrix to perform a SpMM operation Z = S • V to get the final result ( 6 7 ).

C. SDDMM Method
CPSAA can use the mask matrix to convert the DDMM operation to SDDMM operation.However, current SDDMM solutions such as DOTA [23] and SANGER [19] can hardly extend to ReRAM-based sparse attention for the following reasons.1) Vector-wise unmatched with matrix-wise.DOTA and SANGER adopt vector-wise parallelism to utilize the spatial locality and reduce the random memory access, i.e., dividing the matrix into single vectors for processing.ReRAM naturally has matrix-wise parallelism (spatial locality failure).Thus, it is not feasible to map a vector-wise approach to matrix-wise hardware.2) Less flexible matrix storage in ReRAM.The input matrix Q and K can both be reordered and scheduled in DOTA.However, CPSAA can only reorder and schedule Q because the K matrix must be fixed in the crossbar arrays.The failure of scheduling K makes the overall scheduling methods of DOTA and SANGER fail in ReRAM.3) Difficult to remove all random memory access.The SDDMM scheduling methods of DOTA and SANGER cannot eliminate the random memory access because they cannot access every non-zero value without access zero-values.Therefore, they can never achieve their ideal performance when extending to ReRAM.Three main challenges led to no available ReRAM-based SDDMM method, i.e., vectors binding, fixed K, and random memory access.
Here we use the 4×128 Q and K matrices as the example to explain our SDDMM acceleration method.As Figure 8 (b) shows, the elements of the Q matrix are marked from 'a' to 'p', where 'a' is a row vector with 32 32-bit numbers.The most popular ReRAM-based mapping strategy (used in ReBERT and ReTransformer) of K T uses 32 arrays to store each bit of 32-bit K T (one array for one bit K T ).This mapping binds all vectors of K T and makes it impossible to utilize the vectorwise parallelism.So we map all bits of one vector into the same ReRAM array as Figure 8  ReRAM-ReCAM Coupling.We assume that the 4×4 matrix shown in Figure 8 (a) is the sparse mask matrix stored in a ReCAM array.In the beginning, the ReCAM array will perform the row-wise searching row-by-row as the colored arrows in Figure 8 (a) shows.We are taking the second row (green arrow line) in Figure 8 (a) as an example.The matching result of the second row indicates that the second row of S has two zero elements (white cells), so we need to avoid the calculation of these zero-value cells.To locate the address of non-zero elements, the coordinates ⟨α, β i ⟩ of these matched '1' cells will be sent to the CTRL.Then, the CTRL will utilize the row coordinate α to find the corresponding row of Q as the green arrow in Figure 8 (b) shows.At the same time, the CTRL will utilize the column coordinate β i to find the corresponding ReRAM arrays in Figure 8 (e).As these green arrow lines in Figure 8 (e) show, the row indicated by α will be sent to the IR of these arrays indicated by β i .The searching of the ReCAM array will iterate four times, and the other three iterations are marked with red arrow lines in Figure 8 (a).When all iterations are finished, the Q matrix will be distributed to the IR of the ReRAM arrays, like these red and purple rectangles in Figure 8 (e) show.Finally, all arrays can perform the VMM operations using the topmost vector in their IR until all vectors are processed.To visually represent the advantage of our new approach, Figure 8 (d) uses the same color to present the elements which can be parallel calculated.We can see that our new method can finish the calculation of a 4×4 S matrix in two cycles.
Effect of Sparsity.The sparsity of Figure 8 (a) is 0.5, so our novel method can reduce the latency from four cycles to two cycles.Note that the sparsity of the attention mechanism is around 0.1, so our approach can save up to 10× latency in real-world applications.In summary, our new method can finish the calculation of SDDMM with fewer latency iterations and higher parallelism.Further, our method can efficiently utilize the unstructured sparsity without complex scheduler and control signals.

D. SpMM Method
Current Problems.Many ReRAM-based SpMM solutions [17], [28] have been proposed.We classify them into two categories while extending to ReRAM-based sparse attention.One chooses to store sparse S T in ReRAM while using dense V T as the input.Another chooses to store the dense V in ReRAM while using sparse S as the input.A heavy waitfor-write latency occurs if we choose the first one.That is because we can not perform Z = S • V = (V T • S T ) T until we finish the write of S T (the write of S T will become the critical datapath).Figure 9 shows the visualization limitation of the second solution.As set in BERT [4] and A 3 [7], S is a 320×320 sparse matrix and V is a 320×64 dense matrix.With the dense matrix V stored in the ReRAM arrays (Figure 9 (b)), people can perform the VMM operation row-by-row with the sparse matrix S as the input (Figure 9 (a)).Because many elements of S are zero-value, current solutions choose to set zero signals for these zero values.However, these solutions will cause the array wasting problem, as Figure 9 (b) shows.Taking the first row of S (marked in red frame rectangle) as an example, many rows (rows other than red) in ReRAM arrays keep idle.Therefore, the parallelism and the runtime memory utilization of ReRAM is quite low.In addition, the SpMM method in Figure 9 can only save energy but cannot save execution cycles.
Detailed Designs.The sparsity of matrix S is the same as the mask matrix, so it is possible to re-arrange the storage of the V matrix using the mask matrix.Here we also use a 320×320 sparse S matrix and a 320×64 dense V matrix as an example to illustrate the details.As Figure 10 (a) shows, the mask matrix is stored in a ReCAM array (blue cells are '1' and white cells are '0').First, the ReCAM array will perform a row-wise searching operation (red arrow line), which can find the coordinates ⟨α, β i ⟩ of these matched '1' values ( 1 ).Then, the coordinates of these matched cells will be sent to the CTRL, and the CTRL will use the column coordinates β i to find the corresponding rows of the V matrix (red frame rectangle in Figure 10 (b)), which will be stored to the same ReRAM arrays (red lines in Figure 10 (b) 2 ).This mapping phase will iterate to each row of the ReCAM array, and we further show the mapping of the second row (green lines and rectangles) and the last row (yellow lines and rectangles).At the same time, the CTRL will also send the column coordinate β i to the sparse S matrix.The S matrix can find these elements (marked with red cells in Figure 10 (c)) corresponding to β i and send these cells to the corresponding IR (red cells in Figure 10 (b) 3 ).After all vectors are well mapped, CPSAA can simultaneously perform a VMM operation between the vectors in the IR and the V matrix stored in the ReRAM array to calculate all rows of the output Z matrix.
Advantages.Assuming that the sparsity of the S matrix is 0.1, so we need around 320×64 32×32 ReRAM arrays to store the matrix V .Meanwhile, the SpMM operation Z = S •V will be finished in one cycle.Following the traditional mapping shown in Figure 9, 10×64 32×32 ReRAM arrays are required for storage V matrix, and 320 cycles will be taken to generate the Z matrix.The comparison shows that our novel SpMM method saves 320× execution time at the cost of 32× memory space.When memory space is limited, we can process the S matrix in batches.In the case of two batches, we can get 160× execution time at the cost of 16× memory space.

E. Processing Long Sequences
This section describes the memory and time complexity of CPSAA when processing long sequences.We assume the input matrix X has 'T' tokens, and the dimension of each token are set to 'D'.
Memory Complexity.CPSAA has two types of memory, i.e., on-chip buffer and crossbar arrays.Four matrices are stored in the crossbar arrays, i.e., W S , W V , X T , and V .The memory complexity of W S and W V are O(D 2 ).The memory complexity of X T and V are O(D × T ).Four matrices are stored in the on-chip buffer, i.e., X, M, S, and Z.The memory complexity of X, M, and Z are O(D × T ) while the memory complexity of S is O(T 2 ).Therefore matrix S is the only one that has quadratic memory complexity depending on sequence length 'T' Time Complexity.In general, assume we are given two matrices A and B of sizes α × β and β × γ, respectively.The multiplication of A * B requires α * β * γ multipliplyaccumulate operations, and the result matrix C = A * B is of size α × γ.Following the notations in Figure 4 (c), we have S = M ×X T , where M and X are of sizes T ×D.Therefore, the time complexity of matrix S is O(T 2 ).We also have Z = S ×V , and the sizes of S and V are T × T and T × D, respectively.Therefore, the time complexity of matrix Z is O(T × D).

F. Application-level Designs
CPSAA is designed to eliminate the massive off-chip data transmissions in the current sparse attention accelerators.However, real-world NLP applications usually involve several attention layers working with full-connection (FC) layers, such as BERT [4].The encoder is the basic computation unit of BERT, which contains one attention layer and several FC layer.We configure one CPSAA chip that works with several ReRAM-based FC layer to make up one encoder, and several encoders work together to make up the BERT-based solution.
We perform a pre-training and fine-tuning model to get the weight matrices, W Q , W K , and W V , for a BERT-based text classification task [4].Then, we pre-calculate W S = W Q • W T K and QU(W S ).We pre-store the W S , W V , and QU(W S ) matrices in some ROAs.Note that all the above procedures are preprocessing because these matrices can be reused for different input sequences.In the beginning, an input matrix X arrives at the Input Buffer (IB).First, the X matrix will be processed by the CPSAA with no off-chip memory access to generate the Z matrix.Second, the Z matrix will be sent to the FC layer to generate the result of this encoder, which is processed following the ReRAM-based dot-product method proposed in ISAAC [27].Finally, the result of the previous encoder will be sent as the next encoder's input until the final inference results are obtained.

V. METHODOLOGY
CPSAA Configurations.We employ a Python cycleaccurate simulator to model the CPSAA attention accelerator.We use the 1000GB/s On-Chip Interconnect (OCI) for the onchip transfer bandwidth [9].ReRAM's energy consumption is obtained with 7pJ per bit as in [35].As configured in ISAAC [27], the "cycle" in CPSAA means the time of ADC processing 32 column signals, i.e., 25ns.
The CPSAA configurations are shown in Table II.We configure CPSAA with 64 tiles, and each tile includes one group of peripheral components, 11 ROAs, and 56 WEAs.We use two 512×512 ReCAM arrays as the Scheduler to store the mask matrices, while we use 32×32 crossbar as the ReRAM array to perform the VMM operation.The crossbar configurations used in CPSAA are designed under the 32nm process with 533MHz clock frequency [21].Based TaOx ReRAM cells from [21], we conduct SPICE simulation for the area and power of crossbar configuration.We use CACTI 6.5 [20] in 32nm technology to evaluate the power and area of all registers.To obtain parameters of other peripheral components, we use Cadence-simulator [13] for S/H, S+A in a crossbar.The power, area, and latency of a 2-bit precision DAC and a 2.0 mW 8b 1.0 GS/s ADC in 32nm CMOS are obtained from [26] and [14], respectively.The crossbars are read and written in a row parallel manner, and the SET/RESET latency is 1.52/2.11ns for single-level cells (SLC) [34].The area and energy of SU, BU, QU, DQU, and CTRLs are established by SPICE circuits, too.The write endurance of the ReRAM cell can be alleviated greatly.For instance, considering up to 10 12 ReRAM write endurance [39], CPSAA can achieve hundreds of millions (10 8 ) of inferences when processing millions of tokens documents (10 4 times writing).
As configured in Transformer [30], BERT [4], A 3 [7], and SANGER [19], we set the model dimension d model = 512 and d K = d Q = 64 for all workloads.In addition, model dimension (d) is not likely to vary widely since a choice of too large d can lead to a decrease in model accuracy [37].To fit the memory space of CPSAA, we divide all datasets into some little batches, and each batch has 320 embeddings, as set in BERT [4] and A 3 [7].Embeddings in the same batch can be parallel processed by CPSAA without off-chip data transmission, and embeddings in different batches are processed in serial with small off-chip data transfers.We use the Giga Operations per second (GOPS) as the performance metric (throughput) and the throughput per Watt (GOPS/W) as the energy metric (energy efficiency).
Data Overflow Prevention.The precision of input and weight matrices are set to 32-bit.The learning rate for GLUE and SQuAD v2.0 datasets are 2e-5 and 3e-5, and the number of fine-tuning epochs is set to 3 and 2, respectively.The attention mechanism involves lots of matrix multiplication calculations, so preventing data overflow is as important as ensuring accuracy.We extract the Exponential Bit (EB) of the whole array to make the Fraction Bit (FB) remaining 32-bit fixed-point as in [5].Thus, all VMM operations in CPSAA can be performed between 32-bit fixed-point numbers.After we get the results of the VMM operation between the FB, we can multiply the EB by the FB and get the final results.
Comparison Platforms.CPSAA is compared with stateof-the-art platforms as follows.
We choose GPU as the NVIDIA TITAN RTX GPU@1770MHz, 576 Tensor cores, 4608 CUDA cores, 24GB memory, 672GB/s memory bandwidth, and 280 Watt TDP.We choose Bigbird as the attention algorithm for the GPU [38].We measure power consumption via nvidia-smi and execution latency using CUDA Event.We measure the performance of GPUs using PyTorch with cuBLAS 11.2.We choose the FPGA-based software-hardware co-design accelerator proposed in [40] as the FPGA platform for comparison.The configurations are also following [40].We choose the configurations of SANGER as the ASIC-based attention accelerator for comparison [19].We choose two architectures, ReBERT [11] and ReTransformer [36], as the PIM-featured attention accelerator for comparison.The configurations of crossbars used in ReBERT and ReTransformer are the same as CPSAA.

A. Performance and Energy Efficiency
Figure 11 (a) and (b) present the average speedup and energy efficiency of different models, respectively.Because BERT, GPT-2, and BART are all based on encoders and decoders, while the current sparse attention accelerators can accelerate both the encoders and decoders.Therefore, the current sparse attention accelerators are effective in accelerating all models.The speedups of these three models have the same trend while the specific values vary by 10%.To make the experimental results more concise, we present the results of BERT model to reveal the differences between CPSAA and other platforms.
CPSAA against GPU and FPGA.As shown in Figure 12, the average throughput of GPU and FPGA platforms are 102 GOPS and 284 GOPS.Compared with the average 9142 GOPS of CPSAA, CPSAA has 89.6× and 32.2× performance improvement vs. the GPU and FPGA platforms, respectively.As for the energy efficiency shown in Figure 13, GPU, FPGA, and CPSAA have an average of 0.63 GOPS/W, 8.6 GOPS/W, and 476 GOPS/W energy efficiency, respectively.Therefore, CPSAA can save 755.6× energy and 55.3× energy compared with GPU and FPGA platforms.CPSAA can achieve far better Fig. 13.Comsumed energy normalized to CPSAA performance and energy-saving than GPU and FPGA platforms for the following reasons.First, CPSAA is a PIM-based platform, which can save lots of execution time by eliminating the massive off-chip random memory access.As shown in § II-D, these off-chip data transmissions will take up to 60% latency.Second, CPSAA elegantly exploits the high parallel VMM operation of ReRAM arrays by developing two sparse attention-specific methods.Compared with GPU-and FPGAbased platforms, these ReRAM-based matrix multiplication methods can achieve higher in-situ processing parallelism.CPSAA against SANGER.As Figure 12 and Figure 13 show, SANGER has an average of 513 GOPS throughput and 22.4 GOPS/W energy efficiency.Therefore, CPSAA has an average of 17.8× throughput improvement and 21.3× energysaving compared with the ASIC-based platform SANGER.SANGER proposes a prediction-based pruning method to reduce the unnecessary calculations of the attention mechanism.SANGER also presents the "splitting and packing" method to reduce the random memory access.The above optimizations make SANGER achieve 5.03× and 1.81 performance improvement compared with the GPU-and FPGAbased sparse attention accelerators.However, there are some unsolved issues left by SANGER, which CPSAA solves to achieve further speedups.First, SANGER adopts a softwarebased pruning method to generate its mask matrices, which has full data transfers between the memory and the offchip processor.By proposing a PIM-based pruning method, CPSAA can generate the mask matrix while avoiding off-chip data transfers.Second, SANGER design the pruning phase to work serial with the attention calculation phase, while CPSAA can perform the pruning phase parallel with the attention calculation.Finally, CPSAA can directly and efficiently exploit unstructured sparsity by coupling ReCAM arrays to generate control signals for VMM operations, which can avoid the pretty high "splitting and packing" overhead and the processing elements reconfiguration overhead in SANGER.

N o r m a l i z e d E n e r g y
CPSAA against ReBERT and ReTransformer.The throughput and energy efficiency of ReBERT and ReTransformer are also shown in Figure 12 and Figure 13.Re-BERT and ReTransformer have 2696 GOPS and 2381 GOPS throughput, respectively.Therefore, CPSAA achieves 3.39× and 3.84× performance improvement vs. ReBERT and Re-Transformer.ReBERT and ReTransformer have an average of 83.7 GOPS/W and 97.1 GOPS/W energy efficiency, respectively.Thus, CPSAA has an average of 5.7× and 4.9× energy saving compared with ReBERT and ReTransformer.We also analyze the reasons for these results as follows.
ReBERT and ReTransformer are both PIM-based platforms, and they can achieve performance improvement against the GPU-and FPGA-based platforms since they do not contain the off-chip data transfers.Compared with ReBERT and ReTrans-

B. Effectiveness of Calculation Mode
This work proposes a new calculation mode to hide the ReRAM write overhead while improving parallelism by removing data dependency.Therefore, we also design experiments to evaluate the effectiveness of our calculation mode, as shown in Figure 14.We design a dense-version of CPSAA, called CPDAA, to eliminate the acceleration effect of the sparsity.Figure 14 shows the execution time and energy consumption of ReBERT, ReTransformer, and CPDAA, which is normalized to the execution time of CPDAA (CPDAA-T) and the energy consumption of CPDAA (CPDAA-E).It is shown that ReBERT and ReTransformer take 1.31× and 1.64× execution time against CPDAA, respectively.As for the energy consumption, ReBERT and ReTransformer consume 1.30× and 1.21× than CPDAA.To further reveal the reasons for these results, we design more experiments as follows.
We further design two metrics to compare the calculation mode of CPDAA with ReBERT and ReTransformer, i.e., the execution time of waiting for write (CPDAA-W4W) and the number of arrays for parallel VMM operation (CPDAA-P).The experimental results in Figure 15 are all normalized to ReTransformer (ReTran-W4W and ReTran-P).ReBERT and CPDAA take 1.94× and 1.48× execution time on write operations compared with ReTransformer.In contrast, the VMM parallelism of ReBERT and CPDAA is 2.88× and 2.03× compared with ReTransformer.ReTransformer has the minimal write latency but the worst VMM parallelism because it is designed to reduce the write overhead as much as possible.However, they turn the VMM operations that could be executed in parallel into strictly serial execution, reducing the wait time for write operations while increasing the wait time for the previous VMM operations.ReBERT has the maximal write overhead but the best VMM parallelism because they use a write-then-calculate computational mode, which maximizes the VMM execution efficiency but takes longer to wait for write.CPDAA takes a trade-off between ReBERT and ReTransformer, which reduces unnecessary write overhead while preserving necessary VMM parallelism.
In the experiments shown in Figure 15, ReTransformer optimizes the write operations, but the performance of which is worse than ReBERT for the following reason.Taking into account the non-negligible write overhead, this work designs all ReRAM-based platforms to use a single-level cell (SLC) with lower write latency and lower write energy consumption.If a multi-level cell (MLC) with higher write overhead is used, the advantage of ReTransformer will be revealed.

C. Acceleration of Pruning Architecture
CPSAA proposes a novel ReRAM-based PIM-featured pruning method to generate the mask matrix.Therefore, we also develop experiments to evaluate the effectiveness of our novel pruning architecture.As Figure 16 shows, we choose five indicators to reveal the advantages of our novel method against the state-of-the-art pruning method proposed in SANGER.They are the execution time of the pruning phase (Pruning-T), the execution time of the attention calculation phase (Attention-T), the number of pruning phase's VMM operations (VMM-N), the CTRL runtime scheduling time (CTRL-T), and the accuracy comparison (Accuracy).The results in Figure 16 are all normalized to CPSAA, i.e., the multiple of SANGER divided by CPSAA.
The pruning phase in SANGER takes 85.1× execution time compared with our novel pruning method.That is because our novel method eliminates the off-chip data transfers and greatly improves the computational parallelism when generating mask matrices, which can save lots of execution times.Moreover, the Attention-T of SANGER is 18.7× of CPSAA, which comes from two aspects.First, CPSAA eliminates massive off-chip data transfers compared with SANGER.Second, CPSAA can leverage the high parallel VMM operations of the ReRAM arrays, which can greatly reduce the latency of the VMM operations.Figure 16 also shows that CPSAA can save around 16.37× VMM operations when generating the mask matrix.CPSAA can generate the mask matrix directly using the input matrices, while SANGER needs lots of VMM operations to generate some intermediate matrices Q and K first.
Experimental results show that SANGER takes 11.4× execution time on CTRL-T when compared with CPSAA.That is because SANGER's "splitting and packing" algorithm has complex scheduling control signals for sparse S and pretty high processing elements re-configuration overhead.CPSAA can use ReCAM Scheduler to reduce the control signals and eliminate the re-configuration overhead by designing a ReRAM-ReCAM coupling SDDMM and SpMM methods.CPSAA also adopts a new equation using the low-precision weight matrix W S and quantified M to generate the mask matrix.Therefore, it is necessary to evaluate the accuracy of our pruning method.Figure 16 shows that CPSAA loses less than 0.2% of accuracy compared to SANGER.

D. Novel SDDMM and SpMM Designs
Compared to current PIM-based dense attention accelerators ReBERT and ReTransformer, CPSAA designs two novel ReRAM-based methods to accelerate the SDDMM and SpMM operations, respectively.Therefore, we also design experiments to reveal the effectiveness of the novel SDDMM and SpMM methods.As shown in Figure 17, the metrics of this evaluation are the execution time and the consumed energy.The baseline is the execution time of the DDMM operations (DDMM-T) and the consumed energy of the DDMM operations (DDMM-E) in ReBERT.All experimental results are normalized to DDMM-T (100) and DDMM-E (100).The latency of the SDDMM and the SpMM methods are 17.5% and 0.54% of the DDMM method, respectively.The reason for the speedups of the SDDMM operation is as follows.The SDDMM approach can minimize these unnecessary VMM calculations and greatly reduce the latency by using ReCAM Scheduler to guide the generation of control signals.Our new SpMM method can significantly reduce the number of idle rows in VMM operations, greatly increasing the parallelism and reducing the latency.Figure 17 also shows that the consumed energy of the novel SDDMM and SpMM methods are 32.9% and 25.2% of the DDMM method.It is easy to understand that the SDDMM method can save energy by using the mask matrix to avoid computing unnecessary VMM operations.The novel SpMM method can save energy by avoiding a large number of idle rows of the ReRAM array.
We also design experiments to reveal how the proposed SDDMM performance is affected by the crossbar size, as Figure 18 (a) shows.The X-axis indicates the crossbar size, and the Y-axis indicates the average (average of running all datasets) speedup of our SDDMM method vs. the ReRAMbased DDMM solution.We can find that the speedups of the SDDMM method decrease as crossbar size increases.That is because the key idea of our SDDMM method is to convert the matrix-wise parallelism of the ReRAM array to vector-wise parallelism.As the crossbar size arise, more vectors can be stored in the same array, and the vector-wise parallelism will decrease.Based on the above phenomena, we recommend using an array size that matches the value precision to maximize vector-wise parallelism.
In Figure 18 (b), we design experiments to reveal the runtime memory utilization, throughput, and data replication between the SpMM method in Figure 9 and Figure 10.SpMM-B refers to the baseline SpMM method in Figure 9 and all its runtime memory utilization, throughput, and data replication are normalized to 1. SpMM-M, SpMM-T, and SpMM-R refer to the runtime memory utilization, throughput, and data replication of our SpMM method.For all five datasets, our SpMM method has an average of 9.36× runtime memory utilization improvement, 298× throughput improvement, and 30.4× data replication overhead.These experimental results are consistent with the CPSAA design principle of using space cost in exchange for low SpMM latency.

E. Scalability
First, we study the scalability concerning the data volume of input embeddings on the WNLI dataset.Because we cannot get datasets with multiplied data volumes in real-world datasets, we randomly select 1/16 to 1/2 input embeddings in the WNLI dataset to evaluate the effect of the data volume.Figure 19 (a) shows the throughput of GPU, FPGA, SANGER, ReBERT, ReTransformer, and CPSAA processing various sizes of the WNLI dataset.The throughput of CPSAA can be maintained at a relatively stable value.That is because we divide the input embeddings into small batches with 320 sequences, and different batches are processed serially.Therefore, a larger dataset has more batches and takes more execution time, making the number of Giga operations per second (GOPS) remain stable.
We study the scalability concerning the number of encoder layers in BERT.We set the number of the encoder from two layers (2L) to 32 layers (32L), and we choose the GPU baseline as the comparison platform.As Figure 19 (b) shows, the throughput of GPU-based platforms declines noticeably when the number of encoder layers increases.Unlike the trend in the GPU platform, the throughput of CPSAA remains stable.That is because more encoder layers will introduce more computational tasks, generate more intermediate matrices, and require more memory space.Therefore, the GPUbased platform has more random memory access while taking more execution time.But in the PIM-based platform such as CPSAA, more memory space means more computational resources, and the throughput of CPSAA will not increase as the number of encoder layers increases.
We design experiments to reveal the sequence lengths scalability.We configure six sequence lengths, i.e., 128, 256, 512, 1024, 2048, and 4096.Figure 20 shows the average throughput of running all datasets.The three von Neumann accelerators, GPU, FPGA, and SANGER, show the same trend of sequence length scalability.The CPU-, FPGA-and SANGER-based accelerators can maintain stable throughput when the sequence lengths ≤512.When the sequence lengths are greater than 512, the throughput of the above platforms decreases significantly as the sequence lengths increase.That is because the increased sequence length generates a large amount of intermediate data, which exacerbates the random access and off-chip transfers of the von Neumann accelerators, thereby reducing system efficiency.The above results are consistent with the time complexity analysis in Section IV-E.The throughput of the PIM-based accelerators, ReBERT, ReTransformer, and CP-SAA, do not vary significantly with sequence lengths.That is because the PIM-based accelerators can on-chip process all the intermediate data without transferring them to the main processor.

VII. CONCLUSION
We investigate current sparse attention accelerators and find the reasons for their limited speedups.This work presents a crossbar-based PIM-featured sparse attention accelerator, CPSAA, to tackle current problems.We design a novel calculation mode of attention mechanism to hide the write overhead while maintaining the VMM parallelism.We also present a new PIM-based pruning method to eliminate the off-chip memory access when generating the mask matrix.To solve the challenges that all current ReRAM-based attention accelerators can hardly efficiently extend to sparse attention, we design novel ReRAM-based SDDMM and SpMM methods.Finally, we design experiments to evaluate the effectiveness of all designs proposed in this work.Our experimental results show that in performance and energy-saving, CPSAA outperforms state-of-the-art accelerators, ranging from GPU, FPGA, ASIC, and PIM.The experimental results also reveal the effectiveness of all these core designs mentioned in this work.

Fig. 3 .
Fig. 3. Response time breakdown of SANGER and DOTA attention accelerators in TableI.These solutions use high parallel ReRAM arrays to significantly reduce the latency of DDMM operations[11],[36].However, since the sparsity of the attention matrices is not considered, current PIMbased attention accelerators need to calculate all five DDMM operation steps, wasting lots of computational resources on irrelevant tokens.Therefore, how to design a PIM-based sparse attention accelerator and reducing unnecessary calculations become the key to achieve further accelerations.

Fig. 4 .
Fig. 4. The calculation mode of (a) ReBERT, (b) ReTransformer, and (c) dense-version CPSAA smaller than the attention calculation since the pruning phase of SANGER and DOTA uses low-precision computing.Second, most of the overhead of the mask generation phase comes from the off-chip memory access.The MA-GE-M takes an average of 94.6% (92.7% in DOTA) response time while the MA-GE-P takes 5.4% (7.3% in DOTA).The reason is that the MA-GE of SANGER (DOTA) needs to access Q and K from the off-chip memory, which involves many off-chip data transfers and takes lots of response time.Third, the attention calculation also spends a lot of time on off-chip data transfers.The AT-CA-M takes an average of 71.2% (63.5% in DOTA) response time while the AT-CA-P takes 28.8% (36.5% in DOTA).That is because all the input matrices, Q, K, V , and S, need to be sent from the off-chip memory to the ASIC-based processor.These off-chip data transmissions greatly increase access latency and further hurts performance.The massive off-chip data transmissions motivate us to design a novel sparse attention accelerator to simultaneously accelerate MA-GE and AT-CA with pretty low off-chip random access.

Figure 5
Figure5shows the overview architecture of CPSAA, which contains several Tiles, and each Tile has two main parts, i.e., the peripheral components (red-dotted rectangle B ) and ReRAM arrays.The peripheral components (PC) contain a controller (CTRL), several Buffers, Quant and De-Quant Units (QU and DQU), one Softmax Unit (SU), one Binarization Unit (BU), and two ReCAM arrays worked as Scheduler.The ReRAM arrays are classified as read-only arrays (ROA) and write-enable arrays (WEA).The parameters

Fig. 6 .
Fig. 6.(a) Details of QU, (b) details of SUobtained from pre-training are evenly distributed to the readonly arrays in each tile of CPSAA.The tiles assigned to the same attention head communicate with each other through on-chip bus.Because real-world applications usually need multiple attention layers working together, we also show the input matrix from the previous layer and the output matrix to the next layer, which is the off-chip data transmission managed by the data transfer controller (DTC A ).We use the circled color numbers to show the CPSAA dataflow, which is corresponded to the numbers in Figure7.The functions of all components are as follows:CTRL.The controller will generate control signals for all components in one Tile.Buffers.The Input Buffer (IB) will store the input embeddings, i.e., X matrices.The Crossbar Buffer (CB) will store the matrices generated at running time.The address information table (AIT) will record the address information of matrices in the ReRAM arrays, which will be useful when we need to know the location of a specific sub-matrix.QU and DQU.The Quant Unit will quantize the matrix to a low-precision representation (via QU(•) function), and the Dequant Unit will convert the low-precision representation back to high precision (via QU −1 (•) function).The architecture of the QU is shown in Figure6(a), and the DQU is designed in the inverse process.SU and BU.The Softmax Unit will perform the softmax function, and the detailed architecture of the SU is the same as A 3[7], shown in Figure6(b).The input of SU is matrix S, which is calculated by multiple crossbar arrays in one tile.While crossbar arrays calculate matrix S in different tiles, CPSAA will gather these results to generate S. The Binarization Unit will convert a given matrix to the '01' matrix, which is performed by a binarization comparator.ReCAM Scheduler.The ReCAM arrays will store the mask matrices, which will be used to generate control signals for the ReRAM-based VMM operations.

Fig. 11 .
Fig. 11.(a) Average speedups of various models, (b) Average energy efficiency of various models

E n e r g y E f f i c i e n c
r a n s f o r m e r C P S A A E n e r g y -e f f i c i e n c y

Fig. 17 .Fig. 18 .
Fig. 17.Compared the SDDMM and SpMM methods with the DDMM methods in ReBERT, which is normalized to the DDMM operation

Fig. 19 .Fig. 20 .
Fig. 19.(a) Throughput with different size of datasets, (b) throughput with different encoder layers 15. Compared CPDAA with ReBERT and ReTransformer, which is normalized to ReTransformer former, CPSAA designs efficient ReRAM-based SDDMM and SpMM methods, which can save lots of execution time by avoiding these unnecessary VMM operations.Moreover, the novel attention calculation mode of CPSAA can hide the pretty high ReRAM write overhead while maintaining the VMM parallelism.Although CPSAA introduces an extra sparsity pruning phase, it can work parallel with the attention calculation and not introduce extra latency.