Valhalla - distributed information architecture

We have designed a distributed open source framework that allows us to crawl, parse and index code and content. The embedded information retrieval system and the resulting search engine are independent from any single point of failure, central huh, sewer or database and allow the visualization and monitoring of site information architectures on any graphical operating system. The distributed Java clients, which are capable of communicating and self-organizing, replace the traditional server based search engines and employ several methods and technologies that allow the creation of a very efficient, fast, and fault tolerant system. Each client coordinates his tasks with other neighboring clients, retrieves the information, storm the result in a local database, and exchanges the preprocessed data and results with corresponding applications. The paper describes the structure, technology and scalability of our system, provides a performance evaluation. The paper will further describe the techno-political implications of our framework in an environment that is currently defined by political, commercial and security considerations. The framework we have designed is capable of circumventing these harriers. The paper will also discuss the incentive structures and advantages that our application provides not only to site owners hut also to users of our clients. Special attention will be devoted to performance and quality criteria of our framework compared with traditional information retrieval services.