Scalability issues for high performance digital libraries on the World Wide Web

We investigate scalability issues involved in developing high performance digital library systems. Our observations and solutions are based on our experience with the Alexandria Digital Library (ADL) testbed under development at UCSB. The current ADL system provides online browsing and processing of digitized maps and other geospatially mapped data via the World Wide Web (WWW). A primary activity of the ADL system involves computation and disk I/O for accessing compressed multi resolution images with hierarchical data structures, as well as other duties such as supporting database queries and on the fly HTML page generation. Providing multi resolution image browsing services can reduce network traffic but impose some additional cost at the server. We discuss the necessity of having a multiprocessor DL server to match potentially huge demands in simultaneous access requests from the Internet. We have developed a distributed scheduling system for processing DL requests, which actively monitors the usages of CPU, I/O channels and the interconnection network to effectively distribute work across processing units to exploit task and I/O parallelism. We present an experimental study on the performance of our scheme in addressing the scalability issues arising in ADL wavelet processing and file retrieval. Our results indicate that the system delivers good performance on these types of tasks.