Linguistic-Aware WordPiece Tokenization: Semantic Enrichment and OOV Mitigation

The LinPair approach introduces an approach to tokenization, tackling challenges in processing textual data for machine learning applications. This method refines the Word-Piece Tokenization technique by incorporating linguistic insights into token-tag pairs, thereby enriching Large Language Models (LLMs) with syntactic information. LinPair Tokenization is articulated through two distinct strategies: CompleteLinPair Tokenization, which is tailored for smaller datasets and enriches each token with part-of-speech (POS) information, and SmartLinPair Tokenization, designed to mitigate the impact of out-of-vocabulary (OOV) occurrences. The experimental analysis indicates that SmartLinPair Tokenization improves text classification tasks with BERT, showing an increase in F1-Score of up to 13.9% compared to a WordPiece baseline. The CompleteLinPair Tokenization method results in a decrease in classification performance. It is important to note that these findings are experiment specific and may not be generalizable to other contexts.