Analysis and improvement of HITS algorithm for detecting Web communities

We discuss problems with the HITS (Hyperlink-Induced Topic Search) algorithm, which capitalizes on hyperlinks to extract topic-bound communities of Web pages. Despite its theoretically sound foundations, we observed that the HITS algorithm has failed in real applications. In order to understand this problem, we developed a visualization tool LinkViewer, which graphically presents the extraction process. This tool helped reveal that a large and densely linked set of unrelated Web pages in the base set impeded the extraction. These pages were obtained when the root set was expanded into the base set. As a remedy to this topic drift problem, prior studies applied a textual analysis method. We propose two methods which only utilize the structural information of the Web: 1) the projection method, which projects eigenvectors on the root subspace, so that most elements in the root set will be relevant to the original topic; and 2) the base-set downsizing method, which filters out the pages without links to multiple pages in the root set. These methods are shown to be robust for broader types of topic and low in computation cost.