Learning to Rank: Performance and Practical Barriers to Deployment in Enterprise Search

In the field of Web Search (WS), Learning to Rank has become the machine learning method of choice for researchers interested in ranking. In addition to WS, ranking is also the primary challenge for engineers when deploying a new Enterprise Search (ES) service within an organization. This paper compares the implementation and performance of traditional, relatively simple methodologies such as BM25 with Learning to Rank and examines the common assumption that whatever method is best for WS must also be best for ES. We experiment on a small manually annotated dataset derived from a ‘real world’ ES service of a large organization. Results indicate that the nDCG@5 performance of one of the traditional methods (BM25-pf) falls just 6.1% short of the Learning to Rank method, with nDGC scores of 0.93 versus 0.99 respectively. Subsequently, we discuss implementation trade-offs between traditional and LTR methods and highlight the fact that traditional methods require far fewer deployment resources, as well as not requiring feature engineering, data preprocessing, or relevance judgements. A further advantage of the BM25 approach is that it does not entail an open-ended commitment for periodic ranking model retraining.