A review on web search engines' automatic evaluation methods and how to select the evaluation method

Nowadays search engines are recognized as the pathway for accessing the tremendous amount of information in the internet. They provide aids and services for solving users' different information needs. Thus, being able to evaluate their effectiveness and performance is constantly gaining importance because these evaluations are useful for both developers and users of search engines. Developers can use the evaluation results for improving their strategies and paradigms in the development of search engines. Users, on the other hand, can identify the best performing search engines and in a better, quicker and more accurate way, gratify their information needs. Evaluation of search engines can be done in two different ways; either manually using human arbitrators or automatically using automatic machinery approaches which do not use human arbitrators and their judgments. In the case of manual evaluation methods, by now numerous and standard activities had been carried out by organizers and participants of conferences like TREC or CLEF. In the case of automatic evaluation methods, unlike variety of efforts which had been done by different researchers, no categorization and organization of such methods exists so far. As a result, anyone that wants to use one of the automatic evaluation methods must read all the relevant literature of these methods which is very time consuming and confusing activity. In this paper, we have reviewed almost all the important reported automatic methods for evaluation of search engines. Analyzing the results of this review, we have stated the requirements and prerequisites of using any of these methods. At the end, a framework for selecting the best pertinent method for each evaluation scenario has been suggested. Instead, automatic methods of evaluation are cheaper and faster to be performed.