On user-oriented relevance judgement

Abstract

Relevance has been acknowledged as a fundamental and centre concept in information science. Not satisfied with the narrow and technical definition of system relevance, researchers turn to the subjective and situational aspect of this concept. How does a user perceive a document as relevant? The literature on relevance has identified numerous factors affecting such judgment. Taking a cognitive approach, this study focuses on the criteria users employ in making relevance judgment. Based on Grice's theory of communication, this paper proposes a five-factor model of relevance: topicality, novelty, reliability, understandability, and scope. Data are collected from a semi-controlled survey and analyzed following a psychometric procedure. The result shows that topicality and novelty are the key relevance criteria. Theoretical and practical implications of this study are discussed.