Investigating sample selection bias in the relevance feedback algorithm of the vector space model for Information Retrieval

Information Retrieval (IR) is concerned with indexing and retrieving documents including information relevant to a user's information need. Relevance Feedback (RF) is an effective technique for improving IR and it consists of gathering further data representing the user's information need and automatically creating a new query. As RF relies on the ability of an IR system to learn new queries and is mostly based on statistical methods, a parallel between RF and statistical Machine Learning (ML) can be drawn. However, the effectiveness of RF is due to the biased selection of the sample data, thus contradicting the requirement that effective statistical learning is based on unbiased sample data. This paper studies this contradiction and suggests that RF cannot be straightforwardly studied within statistical ML without considering the intrinsic nature of the data managed by an IR system and of the user's information need. In particular, the paper reports that an RF algorithm is mostly influenced by the distance between the informative content of the training set and the informative content of the test set and is not influenced by sample selection bias.