An intelligent agent for content-based indexing and retrieval of documents

The amount of information available on the Internet is currently growing at an incredible rate. However, the lack of efficient indexing is still a major barrier to effective information retrieval on the Web. This paper presents the design of an intelligent agent for content-based indexing and retrieval of relevant documents from a large collection such as the Internet. The agent aims at improving the quality of retrieval by capturing the semantics of the documents. It performs the conventional keyword based indexing and introduces a thematic relationship between parts of text using natural language understanding and a linguistics theory called rhetorical structure theory. The agent described in this paper will be implemented and compared against several indexing systems. It is expected to produce a satisfactory improvement over existing techniques.