Focused Crawling for Retrieving E-commerce Information Based on Learnable Ontology and Link Prediction

With the rapid growth of the e-commerce, how to discovery the specific information such as about buyer, seller and products etc. adapting for the online business user becomes a focused issue to the information search engine. Focused crawling is proposed to selectively seek out pages that are relevant to a predefined set of topics without downloading all of the Web. We present a novel approach for building an intelligent focused crawler which deals with evaluating the pagepsilas content relevance to the e-commerce topic by the domain ontology and the hyperlinks connection to the commercial Web pages by link analysis. In the process of crawling, the domain ontology can evolve automatically by machine learning based on the statistics and rules. Experiments have been performed, and the results show that our approach is more effective than the other traditional crawling algorithms, and prevents the topic-drift with higher harvest rate.