An Ensemble-based Regularization Method for Multi-Head Attention

Multi-head attention is applied in Transformer to jointly attend to information from different representation sub-spaces at different positions. In this work, we are inspired by ensemble learning propose a new disagreement term regularization to ensure that the multi-head mechanism works effectively. Specifically, we treat each head as a base learner of emsemble model and make multi-head more diverse by the bias-variance-covariance decomposition. The test results show that our method is quite effective in Multi-30K Enlish⟹German and IWSLT16 Enlish⟹German translation tasks.