Language Model for Statistics Domain

Since transformer has appeared, there were many studies that proposed variants of some representative language models (e.g., Bidirectional Encoder Representations from Transformers (BERT) [1] and Generative Pre-Training (GPT) series [2]). Huge language models are appearing recently (e.g., Chinchilla [3], Megatron LM), whereas there are studies of domain-specific (or language-specific) language models. For example, BioBERT for bio-informatics [4], SwahBERT for Swahili language [5], and FinBERT for financial domain [6]. Without doubt, statistics must be one of the domains with many collected data (e.g., reports of statistics). Pre-trained language model for the statistic domain will probably deliver much performance improvement in down-stream tasks such as industry code classification and job code classification, and more accurate system for the code classification tasks will contribute to better national statistics and taxation. Indeed, many countries are trying to develop such system, and this paper summarizes some relevant findings and provides suggestions to develop language models for statistics domain.