Enterprise text processing: a sparse matrix approach

Documents, both internal and related publicly available, are now considered a corporate asset. The potential to efficiently and accurately search such documents is of great significance. We demonstrate the application of sparse matrix-vector multiplication algorithms for text storage and retrieval as a means of supporting efficient and accurate text processing. As many parallel sparse matrix-vector multiplication algorithms exist, such an information retrieval approach lends itself to parallelism. This enables us to attack the problem of parallel information retrieval, which has resisted good scalability. We use sparse matrix compression algorithms and compare the storage of a subcollection of the commonly used NIST TREC corpus with a traditional inverted index. We demonstrate query processing using a sparse matrix-vector multiplication algorithm. Our results indicate that our approach saves approximately 35% of the total storage requirements for the inverted index. Additionally to improve accuracy, we develop a novel matrix based relevance feedback technique as well as a proximity search algorithm. The results of our experiment to incorporate proximity search capability into the system also indicate 35% less storage for the sparse matrix over the inverted index.