A Novel Centrality-Driven Clustering Approach for Information Retrieval and Question Answering

Information Retrieval (IR) systems are essential for efficiently managing and retrieving relevant information from extensive and expanding collections of documents. However, traditional IR methods face significant challenges, including issues related to scalability, accuracy, and the ability to effectively capture complex relationships between documents. These challenges are especially critical in question-answering systems where the need for effectively addressing user queries. This paper introduces the Smart Clustering PageRank (SCPR) algorithm, aimed at improving the efficiency and effectiveness of both IR and question-answering systems. SCPR addresses the limitations of conventional methods by combining clustering techniques with centrality measures and the PageRank algorithm. Specifically, SCPR utilizes degree centrality, closeness centrality, and eigenvector centrality to cluster related documents based on their connectivity. This approach enables SCPR to capture various dimensions of the underlying document structure and relationships, resulting in more accurate and contextually relevant retrieval. The SCPR algorithm was evaluated on real-world datasets, particularly the Quora dataset in two distinct forms. The results demonstrate that SCPR outperforms established methods such as BM25, BERT, LocalRank, and LeaderRank in terms of both precision and recall. Future research could focus on extending SCPR to handle multimedia content and incorporating user feedback for further refinement.