Adaptive Sparse Attention via Differentiable Sparsity Ratio Learning

With the wide application of the Transformer model in long sequence tasks, the squared complexity of the attention mechanism has become the main bottleneck of computational efficiency. The traditional sparse attention method reduces the computational load through predefined rules, but still has the drawbacks of information loss and difficulty in dynamic adaptation. This paper proposes a differentiable sparse scale learning attention model (MSDF), parameterizing the sparse scale into a learnable probability distribution to achieve continuous optimization of discrete selection. Thereby adjusting the sparsity ratio of each attention head to achieve end-to-end trainability.