Intelligent spider for Internet searching

As World Wide Web (WWW) based Internet services become more popular, information overload also becomes a pressing research problem. Difficulties with searching on the Internet get worse as the amount of information that is available increases. A scalable approach to support Internet search is critical to the success of Internet services and other current or future national information infrastructure (NII) applications. A new approach to build an intelligent personal spider (agent), which is based on automatic textual analysis of Internet documents, is proposed. Best first search and genetic algorithm have been tested to develop the intelligent spider. These personal spiders are able to dynamically and intelligently analyze the contents of the users' selected homepages as the starting point to search for the most relevant homepages based on the links and indexing. An intelligent spider must have the capability to make adjustments according to progress of searching in order to be an intelligent agent. However, the current searching engines do not have communication between the users and the robots. The spider presented in the paper uses Java to develop the user interface such that the users can adjust the control parameters according to the progress and observe the intermediate results. The performances of the genetic algorithm based and best first search based spiders are also reported.