Pattern Based Comprehensive Urdu Stemmer and Short Text Classification

Urdu language is used by approximately 200 million people for spoken and written communications. The bulk of unstructured Urdu textual data is available in the world. We can employ data mining techniques to extract useful information from such a large, potentially informative base data. There are many text processing systems available to process unstructured textual data. However, these systems are mostly language specific with the large proportion of systems applicable to English text. This is primarily due to language-dependent preprocessing systems, mainly the stemming requirement. Stemming is a vital preprocessing step in the text mining process and its primary aim is to reduce grammatical words form, e.g., parts of speech, gender, tense, and so on, to their root form. In the proposed work, we have developed a rule-based comprehensive stemming method for Urdu text. This proposed Urdu stemmer has the ability to generate the stem of Urdu words as well as loan words that belong to borrowed languages, such as Arabic, Persian, and Turkish, by removing prefix, infix, and suffix from the words. In the proposed stemming technique, we introduced six novel Urdu infix words classes and a minimum word length rule to generate the stem of Urdu text. In order to cope with the challenge of Urdu infix stemming, we have developed infix stripping rules for introduced infix words classes and generic stemming rules for prefix and suffix stemming. We also present a probabilistic classification approach to classify Urdu short text. Different experiments are performed to demonstrate the effectiveness and efficacy of the proposed approach. Comparison with existing state-of-the art approaches is also made. Stemming accuracy results demonstrate the adoptability of the proposed stemming approach for a variety text processing applications.