User-Oriented Relevance Judgment: A Conceptual Model

The concept of relevance has been heatedly debated in last decade. Not satisfied with the narrow and technical definition of system relevance, researchers turn to the subjective and situational aspect of this concept. How does a user perceive a document as relevant? The literature on relevance has identified numerous factors affecting such judgment. Taking a cognitive approach, this study focuses on the criteria users employ in making relevance judgment. Based on Grice's theory of communication, this paper proposes a five-factor model of relevance: topicality, novelty, reliability, understandability, and scope. Data are collected from a semi-controlled survey study and analyzed following a psychometric procedure. The result supports topicality and novelty as the key relevance criteria. Theoretical and practical implications of this study are discussed.