Efficient inverted index with n-gram sampling for string matching in Arabic documents

Text search is the basis of countless applications and techniques. It is constrained by space and time resource limitations inherent in different contexts and scenarios. A common approach to minimize the cost of the general search task is to start by the characteristics which are particular to the targeted entity. In this paper, we propose an approximative index-based text searching algorithm that performances can be customized respect to both time/memory user constraints. The main idea is to exploit the uneven distribution of frequencies of letters and n-grams in natural language text, to reduce the index size and the search time, where we store only the less frequent letters and n-grams. Moreover, our technique can also provide to the user the flexibility to choose the tradeoff between index size and query performance. Search time and the index size can be balanced by varying three parameters in our approach. This makes our approach flexible to different settings. The tests described in this paper are driven on an Arabic collection of more than 450 documents and more than 20 million words. Experimental results show that the size of our n-gram inverted index is reduced by up to 40%–85% (with tolerable performance penalties) compared with those of the full n-gram inverted index. Generalization to other languages should be straightforward as long as the underlying statistical property applies.