Adaptive Web Search: Evolving a Program That Finds Information

Search engines contain programs that compare the words in a user's query to the words and phrases in Web pages. This comparison emphasizes relatively rare terms, terms that occur frequently in a page, and terms in prominent positions (such as a page's title), among other textual clues that suggest what the page is about. Although all search engines differ in the ways they determine which Web pages to present to a user, each incorporates a method that its designers hope will be effective. Nonetheless, retrieval algorithms perform inconsistently—some better in one circumstance, others in another--with no way to know in advance which will be most effective. The authors approach retrieval from a learning perspective. Rather than determining how to combine lexical clues beforehand, they infer how this should be done on the basis of users' evaluations of previously viewed documents. Unlike conventional systems, this approach automatically evolves new retrieval programs through genetic programming. It seems particularly effective for users whose need for information remains consistent over weeks or months.