NTCIR workshops: data collection-based evaluation of information retrieval and its challenges

This paper introduces the NTCIR Workshops. A series of evaluation workshops designed to enhance research in Japanese and Asian language text retrieval, cross-lingual information retrieval, and related text processing techniques such as summarization, extraction, etc. Twenty-eight groups from six countries participated in the first workshop and forty-six groups from eight countries have registered for the second. An evaluation workshop usually provides a set of data usable for experiments and unified evaluation procedures for experimental results. Each participating group conducts research and experiments using the data with various approaches. The importance of reusable large-scale standard data collections in IR research has been widely recognized and an evaluation workshop is now recognized as a new style of active research project that facilitates research by providing the data and a forum for ideas and technology transfer. Finally, some thoughts on future directions are suggested.