Freshness of Web search engines: Improving performance of Web search engines using data mining techniques

Progressive use of Web based information retrieval systems such as general purpose search engines and dynamic nature of the Web make it necessary to continually maintain Web based information retrieval systems. Crawlers facilitate this process by following hyperlinks in Web pages to automatically download new and updated Web pages. Freshness (recency) is one of the important maintaining factors of Web search engine crawlers that takes weeks to months. Many large Web crawlers start from seed pages, fetch every links from them, and continually repeat this process without any policies that help them to better crawling and improving performance of those. We believe that data mining techniques can help us to improve the freshness parameter by extracting knowledge from crawling data. In this paper we propose a Web crawler that uses extracted knowledge by data mining techniques as policies for crawling. For this purpose we include a component to collect additional crawling information. This crawler starts by non-preferential crawling. After a few crawling, it trained by using mining techniques on crawling data and then uses policies for preferential crawling to improve freshness time. Our research represented that crawling with determined polices has better freshness than generic general purpose Web crawlers.