Text Normalization on Thai Twitter Messages using IPA Similarity Algorithm

Twitter often contains many noisy short messages. The noisy text are caused by insertion, transformation, transliteration and onomatopoeia. Text normalization is used for solving these noisy text. In this paper, we present the algorithm that can normalize insertion and homophonic transformation words by converting to International Phonetic Alphabet(IPA) and find the most similarity IPA of out-of-vocabulary and IPA of in-vocabulary using Levenshtein Distance. We used Twitter corpus that contained 2,000 twitter messages for evaluating the proposed algorithm. The experiment result illustrated that the proposed algorithm returned an accuracy of 79.03% when compared to dictionary-based normalization of LextoPlus returned an accuracy 24.19%.