Change aware distributed file system for a distributed search engine

Intranet information retrieval is very important for discovering useful knowledge. In this process, the search engine is useful. However, conventional search engines, which are based on a centralized architecture, are not suited for intranet information retrieval because intranet information is frequently updated. Centralized search engines take a long time to collect Web pages by robots. So, we have developed a distributed search engine, the cooperative search engine (CSE), to retrieve fresh information. In CSE, a local search engine located in each Web server makes an index of local pages. And, a meta search server integrates these local search engines to realize a global search engine. CSE takes a few minutes to update the index because each local search engine scans all files in its own disk. We propose a notification mechanism using a change aware distributed file system. As a result, we realize real-time information retrieval.