SOA: A Sparsity-Oriented Activation on Sub-layers of FFN of Transformers

Since the invention of Transformers, attention-based models have been widely used in machine translation, language understanding, and image processing. However, performance of Transformers comes at cost of significant large computation and memory requirements. Sparsification is a feasible solution to the above problems. In this paper, We propose an new approach called SOA that inserts truncated activation functions into sublayers of FFN to introduce new levels of sparsity. The introduction of the new activation functions can reduce the number of active activations and enable sparse matrix multiplications in following FFN layers. To address the problem of the accuracy degradation caused by the inserted activation functions, we employ fine-tuning to retain accuracy. Our experiments show that for BERT on GLUE datasets, with almost no accuracy degradation(average 0.045%), SOA can produce the average sparsity 60%,and reduce 31.5% of total computation for the entire Transformer, and can save 19% memory space by using run-length coding.