On Modifying Evaluation Measures to Deal with Ties in Ranked Lists

Evaluation metrics for search and ranking systems are generally designed for a linear list of ranked items that does not have ties. However, ties in ranked lists arise naturally for certain systems or techniques. Evaluation protocols generally arbitrarily break ties in such lists, and compute the standard metrics. If the number of ties is non-trivial, it would be more principled to use modified, tie-aware formulations of these metrics. For most commonly used metrics, McSherry and Najork [5] present modified definitions that are tie-aware, and therefore, more appropriate for assessing the quality of systems that retrieve multiple distinct results at the same rank. This paper proposes a tie-aware version of Hit@k that we call ta-Hit@k. Hit@k is also a common evaluation measure that is widely used for some tasks, but is not covered in [5]. We also empirically compare the values of ta-Hit@k and Hit@k for a single example system on a standard benchmark task. CCS CONCEPTS • Information systems → Presentation of retrieval results.