An autonomous system-based distribution system for web search

The tremendous growth in computer networks and storage has fueled the explosive growth of the web. The amount of information accessible from the web has dramatically increased by several orders of magnitude in the last few years, and shows no signs of abating. Developers of web search engines are confronted with the consequent information overload problem. Although many algorithms are widely used to optimize the performance of web search engines, it is still difficult to decrease the updating cycles of the web data for a traditional general-purpose search and indexing system. In this paper, an autonomous system (AS) based solution is presented. One of its main merits is to limit the network load inside an AS. Since AS is the element of the Internet interconnection and a uniformed route strategy is adopted inside an AS, it will be helpful to solve global network overload problem. At the other side, updating cycles are greatly decreased since the distributed system is working in a parallel way at different ASes. Some key algorithms and techniques are put forward and tested for management and synchronization of the whole system. The effectiveness of the solution is verified in laboratory simulation.