Modeling Global-Local Subtopic Distribution with Hypergraph to Diversify Search Results

Search result diversification aims to balance the relevance and diversity of retrieved documents to satisfy the different information needs of users. Three types of approaches have proliferated: explicit models that are based on explicit features (e.g., subtopic coverage), implicit models that are based on implicit features (e.g., the novelty of documents), and ensemble models that utilize both implicit and explicit features. However, the subtopics used by most explicit and ensemble models are usu-ally mined from queries (e.g., using Google Search Suggestions), which may not match the subtopics covered by the candidate documents. Besides, the implicit features used by most implicit models are formulated as either the similarity of documents or the intent of documents. The former cannot directly reflect the relationships of documents at the subtopic level, while the latter cannot capture non-pairwise relationships among documents. To tackle these issues, we propose a novel model that dynamically mines subtopics from the candidate documents and leverages thehypergraph structure to model the diversity of candidate documents, named HGDIV. Specifically, we dynamically mine subtopics from the candidate documents, rather than mining subtopics from queries or using static subtopics as existing methods do. More importantly, we introduce the hypergraph structure to model the diversity of candidate documents for search result diversification, which can capture the non-pairwise relationships among documents. Furthermore, we innovatively model the global and local subtopic distributions to extract the diversity of candidate documents. Experimental results on the public diversity benchmark TREC datasets demonstrate the superiority of our model over state-of-the-art models.