A Semantic- Driven Multimodal Model for Chinese Spelling Checking

Chinese Spelling Checking (CSC) is a task to detect and correct spelling errors in Chinese text, which is important for natural language processing applications. Previous works on CSC have used pre-trained language models and multimodal information to improve the performance. However, most existing CSC models do not fully exploit the rich context knowledge contained in context, nor do they emphasize the role of semantic information for text error correction. In this paper, we propose a semantic-driven multimodal CSC model that can learn the heterogeneous knowledge of speech, vision and meaning from context. Our model consists of three encoders: a semantic encoder, a phonetic encoder, and a graphic encoder. To emphasize the role of semantics in text correction, we feed the output of the semantic encoder to both the phonetic encoder and the graphic encoder, along with the original input. The outputs of the three encoders are fused by an attention mechanism and decoded by a transformer model to generate the spell-check results. We evaluate our model on SIGHAN benchmark datasets. The experimental results show that our model outperforms the state-of-the-art methods on these datasets, especially on correcting long sentences with errors. We also conduct ablation studies and qualitative analysis to show the effectiveness of our model components and the semantic-driven mechanism.