Language model reduction using PCA approach and tackling sparcity using document-weighted smoothing

The Language Model smoothing and reduction of size is an acute problem in information retrieval systems, and has received noteworthy consideration from industry and academia. Despite remarkable progresses in the arena of Language Model, there is no customary tool for selecting important words from Language Model with respect to particular application. A Language Model consists of a number of documents and each document consists of set of words. In this paper, a swift scheme namely LM-Reduction-PCA is proposed to select principal documents from the corpus using Principal Component Analysis (PCA) technique which aids in quick reduction of Language Model size. Moreover, we also propose a new scheme to discover data dependency between the documents of Language Model and sparse words are estimated using document-weighted smoothing technique which assists in achieving the smoothing. The experimental result shows that document-weighted smoothing performs better than without smoothing. Also, a new language model evaluation parameter, total error reported based on out of vocabulary is proposed apart from entropy and perplexity.