By Topic

Freshness of Web search engines: Improving performance of Web search engines using data mining techniques

Sign In

Cookies must be enabled to login.After enabling cookies , please use refresh or reload or ctrl+f5 on the browser for the login options.

Formats Non-Member Member
$33 $13
Learn how you can qualify for the best price for this item!
Become an IEEE Member or Subscribe to
IEEE Xplore for exclusive pricing!
close button

puzzle piece

IEEE membership options for an individual and IEEE Xplore subscriptions for an organization offer the most affordable access to essential journal articles, conference papers, standards, eBooks, and eLearning courses.

Learn more about:

IEEE membership

IEEE Xplore subscriptions

3 Author(s)
Kharazmi, S. ; Dept. of Comput. Eng., WI Lab., Payame Noor Univ., Tehran, Iran ; Nejad, A.F. ; Abolhassani, H.

Progressive use of Web based information retrieval systems such as general purpose search engines and dynamic nature of the Web make it necessary to continually maintain Web based information retrieval systems. Crawlers facilitate this process by following hyperlinks in Web pages to automatically download new and updated Web pages. Freshness (recency) is one of the important maintaining factors of Web search engine crawlers that takes weeks to months. Many large Web crawlers start from seed pages, fetch every links from them, and continually repeat this process without any policies that help them to better crawling and improving performance of those. We believe that data mining techniques can help us to improve the freshness parameter by extracting knowledge from crawling data. In this paper we propose a Web crawler that uses extracted knowledge by data mining techniques as policies for crawling. For this purpose we include a component to collect additional crawling information. This crawler starts by non-preferential crawling. After a few crawling, it trained by using mining techniques on crawling data and then uses policies for preferential crawling to improve freshness time. Our research represented that crawling with determined polices has better freshness than generic general purpose Web crawlers.

Published in:

Internet Technology and Secured Transactions, 2009. ICITST 2009. International Conference for

Date of Conference:

9-12 Nov. 2009