You Might Not Need Attention Diagonals

Pre-trained language models, such as GPT, BERT, have revolutionized natural language processing tasks across various fields. However, the current multi-head self-attention mechanisms in these models exhibit an “over self-confidence” issue, which has been underexplored in prior research, causing the model to attend heavily to itself rather than other tokens. In this study, we propose a simple yet efficient solution: discarding diagonal elements in the attention matrix, allowing the model to focus more on other tokens. Our experiments reveal that the proposed approach not only consistently improves upon vanilla attention in transformer models for diverse natural language understanding tasks, particularly for smaller models in resource-limited conditions, but also exhibits faster convergence in training speed. This effectiveness generalizes well across different languages, model types, and various natural language understanding tasks, while requiring almost no additional computation. Our findings challenge previous assumptions about multi-head self-attention and suggest a promising direction for developing more effective pre-trained language models.