A Hybrid Neural-CRF Framework for Assamese Part-of-Speech Tagging

Natural Language Processing (NLP) involves computational techniques for analyzing and understanding human languages. Part-of-Speech (POS) tagging, a fundamental NLP task, assigns grammatical categories to words in text. The proposed research addresses the task of Part-of-Speech (POS) tagging for Assamese by integrating character-level embeddings into various neural architectures to handle the language’s complex morphology effectively. The study comparatively evaluates Conditional Random Fields (CRF), Hidden Markov Models (HMM), Long Short-Term Memory networks combined with CRF (LSTM-CRF), and Bidirectional LSTM-CRF (BiLSTM-CRF). Leveraging a comprehensive and manually annotated dataset consisting of 13,469 sentences with 213,087 tokens, the research employs the linguistically detailed LDCIL Assamese tagset developed by CIIL Mysore. Experimental results indicate that neural architectures substantially outperform traditional methods, with the BiLSTM-CRF model enhanced by character-level embeddings achieving the highest accuracy of 97.69%. These findings emphasize character-level embeddings’ critical role in elevating the performance of neural network-based POS tagging models compared to traditional CRF methods.