Estimation of Optimal Topic Spider Strategy by Use of Decision Trees

The design of a good topic spider entails an optimal strategy for prioritizing the unvisited URLs. This paper uses a decision tree on anchor texts of hyperlinks to determine the prioritization. A novel taxonomy based topic relevance computation function, which embeds machine learning, classifies pages. Evaluation on different data sets shows that the proposed approach leads to promising results.