Statistical malay part-of-speech (POS) tagger using Hidden Markov approach

Assigning part of speech to running words in a sentence is one of the pipeline processes in Natural Language Processing (NLP) tasks. In this paper, a statistical POS tagger using trigram Hidden Markov Model for tagging Malay language sentences is examined. The problem of the tagger approach is to predict the POS for unseen words in the training corpus that can guess word's POS based on their surrounding information. The predictor has been built based on information of word's prefixes, suffixes or combination of them. Linear successive abstraction has been used for smoothing the probability distribution of part of speech for unknown Malay words given their prefixes or suffixes information. However, for the combination of prefixes and suffixes information, the joint probability distribution has been used. The best performance to predict POS of unknown words are obtained through prefixes information by seeing the first three characters of the words. The accuracy of the tagging is 67.9%. This shows that a statistical tagger for Malay language using Hidden Markov Model is able to predict any unknown word's POS at some promising accuracy.