A new statistical method for performance evaluation of search engines

We present a new statistical method for evaluating search engines' precision performance based on sample queries. The method consists of relevance evaluation and statistical comparison. In relevance evaluation, we present two scoring algorithms: one is a term-based algorithm based on the vector space model, and the other is a new three-level algorithm modeled after manual methods commonly used in information retrieval studies. In statistical comparison, we apply a statistical metric probability of win, in ranking the search engines. Based on a set of sample queries, our method evaluates the relevance of the pages returned by the search engines and compares them statistically In the experiment, our method was applied to three search engines, AltaVista, Google, and InfoSeek, using two query sets derived from the domain of parallel and distributed processing. Our results show that the three-level scoring algorithm with a typical set of parameters obtained results consistent with those obtained using the manual method, whereas the term-based algorithm did not.