Document level assessment for pairwise system evaluation

Topic scores commonly determine the performance of retrieval systems where averaging is performed. Averaging causes loss of unit document scores that should be a measure of system effectiveness. This study aims to use document scores as a unit of measurement in a pair wise system evaluation to overcome the loss of unit document scores. Precision scores of each document per topic are calculated and used in statistical significance tests to determine if a system pair is significantly different. Aggregation of p-values from 50 topics per system pair is performed to determine the system pair significance. These are then compared to significance values from topic scores from average precision. Experimentation shows significance tests using document scores have higher numbers of statistically significant (p <;= 0.01) system pairs compared to using averaged precision scores. Usage of document scores could be an alternative to averaged topic scores in pair wise system comparison.