EA-MLM: Error-aware Masked Language Modeling for Grammatical Error Correction

In recent years, BERT has been used in the task of grammatical error correction (GEC) and achieved good performance. However, few previous studies have investigated the incorporation of real grammatical errors into BERT for the GEC task. We argue that the distribution of GEC data (containing several types of errors) is different from the distribution of BERT pre-training data (usually error-free). To fill this gap, in this paper, we extend masked language modeling and propose a novel error-aware masked language modeling strategy (EA-MLM) to fine-tune BERT so that the representation distribution of the pre-trained BERT is better adapted to the GEC task. We conduct extensive experiments on public datasets and the experimental results demonstrate the effectiveness of our proposed method.