From Complexity to Clarity: Tatabahasa-Centric Lemmatization in Malay Texts

This paper explores the pivotal role of lemmatization tools in Natural Language Processing (NLP) and their significance in enhancing language understanding and text analysis. Through a comprehensive evaluation of various lemmatization techniques, both established and emerging, we aim to improve the precision of NLP applications. Our assessment encompasses diverse linguistic datasets, revealing insights into the strengths and limitations of each approach and guiding the selection of appropriate tools. What differentiates this study is its focus on the substantial advancement accomplished by the rule-based model, which exhibits an impressive 88% precision and outperforms the Sastrawi model by around 3%. Moreover, we introduce innovative strategies to enhance lemmatization outcomes, specifically addressing challenges in multilingual and low-resource language contexts.