Towards a hierarchical scheduling system for distributed WWW server clusters

We present a model for dynamically scheduling HTTP requests across clusters of servers, optimizing the use of client resources as well as the scattered server nodes. We also present a system, H-SWEB, implementing our techniques and showing experimental improvements of over 250%, which have been achieved through utilizing a global approach to scheduling requests. This is the first system to provide a hierarchical scheduling mechanism for distributed HTTP server clusters incorporating dynamic client-server task distribution and distributed data access. H-SWEB uses sophisticated scheduling techniques in monitoring and adapting to workload variation at the client and server clusters for supporting typical digital library tasks, such as fast WWW image browsing. We provide a discussion of our system architecture and implementation, and briefly summarize the experimental results that have been achieved.