Enhancing Multilingual Natural Language Processing with Custom Subword Tokenization: Subword2Vec and BiLSTM Integration for Lightweight and Streamlined Approaches

This research introduces a specialized language tagging model tailored for in-depth word-level analysis of Hinglish, a hybrid language that combines Hindi and English. Leveraging subword embeddings, the model adopts the HinglishTagNet architecture, incorporating Bidirectional Long Short-Term Memory for sequence tagging. The model demonstrates notable accuracy on test sets while maintaining a compact size of 7.55MB, striking a balance between computational efficiency and robust performance. Its compatibility with CPU-based systems reduces dependence on GPUs, providing cost savings without compromising functionality. The model’s streamlined design facilitates efficient data processing while preserving output quality. As a versatile tool, it proves valuable in addressing various natural language processing tasks, demonstrating significance in the analysis of both Hindi and English texts. The reduced model size contributes to an average inference speed of 4ms for 15 tokens on CPU, showcasing practical utility in real-world linguistic contexts.