Handling Chinese OOV with a Combination of Radical-based Sub-words and Glyph Features

In natural language processing tasks, dictionaries must be built and words must be converted into their corresponding embeddings, creating problems such as out of vocabulary (OOV). To solve this problem, algorithms such as byte pair encoding (BPE) have been widely used to generate sub-words from known words and decompose OOV words into existing sub-words. However, Chinese has many types of characters called Kanji, and their distribution in the text is sparse, reducing the performance of BPE algorithms that rely on the frequency of occurrence. Compared with Kanji characters, radicals are less varied and more densely distributed. Moreover, because Kanji characters are hieroglyphs, their glyphs contain a rich amount of information and are unique. In this paper, we take advantage of these characteristics of Chinese and apply the BPE algorithm after converting the characters to the corresponding radicals, and combining sub-word embedding with the glyph feature vector. Experiments show that converting more Kanji characters to radicals results in a more even distribution of characters in the converted Chinese text, and the number of OOV words continues to decrease. Furthermore, at a 30% conversion ratio, the effect of radical-based embedding can outperform the effect of character-level embedding that keeps all Kanji characters unchanged. The results also indicate that combining with embedding using glyph features can improve performance despite the different conversion ratios.