Research on Prototype Framework of a Multi-Threading Web Crawler for E-Commerce

Web crawlers facilitate the search engine's work by following the hyperlinks in Web pages to automatically download a partial snapshot of the Web. Crawling is the initial and also the most important step during the Web searching procedure. A prototype framework of a multi-threading Web crawler for E-commerce application is proposed, in relationship to the former research of search engine. And the design and implementation of a multi-threading Web crawler is described and discussed. The experiment result demonstrates this prototype of Web crawler has better performance.