WikiTND24: A Chinese Text Normalization Database

Labeling large datasets for constructing a text normalization system is cumbersome and time-consuming. Although some self-supervised learning models can reduce the amount of training data needed, labeling sufficient data still requires significant labor and time. To address this issue, we used a self-developed rule-based text normalization system to nor-malize Chinese Wikipedia texts, generating a large amount of training data to develop a non-standard word recognition model. Besides verifying the performance of our model with the FlatTN database test set, we also demonstrated that it can help detect errors in the rule-based system and accelerate the development of a data-driven or machine-learning-based text normalization system. Additionally, we are making this database, WikiTND24, publicly available for everyone to use.