Query subtopic diversification based on cluster ranking and semantic features

Search queries are usually short, ambiguous, and have multiple interpretations. Identifying possible subtopics is one of the key strategy to disambiguate a search query. In recent years, researchers have investigated subtopic mining through a variety of approaches. This paper is aimed at mining a diversified list of subtopics underlying a query. In our approach, first, we apply soft clustering to the subtopic candidates based on frequent phrases to group subtopics of similar intents. Second, we introduce multiple semantic features to rank subtopics in the cluster. The clusters are then diversified by balancing relevancy and novelty. The cluster relevance score is estimated by combining cluster score and subtopic importance. We employ Jaccard coefficient to estimate the novelty of a cluster based on word embedding. Finally, a diversified list of subtopics is generated by selecting top ranked subtopic from each cluster. We conducted experiments on NTCIR-10 INTENT-2 and NTCIR-12 IMINE-2 English subtopic mining test collections. The results conclude that our proposed method outperforms other methods in subtopic mining.