Pre-training with Phonetics and glyphs for Chinese Spelling Correction

Spelling error correction is a challenging task. Extensive approaches nowadays either use rule-based statistics and language models or sequence-to-sequence deep learning methods. In this paper, we propose a new end-to-end CSC model that uses powerful pre-training and fine-tuning methods to integrate phonetic features, stroke features into the language model. The selected tokens are masked according to the confusion set using similar characters, instead of using a fixed token mask as in BERT [10]. in addition to the prediction of the characters themselves, this paper introduces phonetic prediction to learn knowledge of spelling errors at the phonetic level. In addition to this, we add a probability weight in order to balance error detection and error correction throughout the framework. experimental results show that the model achieves a significant improvement over the SIGHAN dataset and outperforms previous state-of-the-art methods.