A Novel Approach to Filter Non-Modified Pages at Remote Site without Downloading during Crawling

The users to search the required information on the Web extensively use the search engines. These, search engines maintain the index of billions of pages for efficient searching. The crawlers of these search engines have to retrieve the pages continuously to keep the index up-to-date. It is reported in the literature that the 40% of the current Internet traffic and bandwidth consumption is due to these crawlers. These crawlers also cause load on the remote server by using its CPU cycles and memory. The authors of this paper address this problem by proposing a novel indexing system based on mobile crawlers. The proposed approach employs mobile agents to crawl the pages. These mobile crawlers identify the modified pages at the remote site without downloading them. The proposed mobile crawler downloads those pages only, which have actually been modified since last crawl. The simulated results of the proposed mobile crawler have shown the reduction in the Internet traffic and load on the remote site considerably.