WWW traffic reduction and load balancing through server-based caching

Research on replication techniques to reduce traffic and minimize the latency of information retrieval in a distributed system has concentrated on client based caching. In this technique, recently and frequently accessed information is cached at a client (or at a proxy thereof) in anticipation of future accesses. Such myopic solutions-focusing exclusively on a particular client or set of clients-will likely have a limited impact. Instead, the author offers a solution that replicates information on a global supply and demand basis. The author proposes a data dissemination mechanism that allows information to propagate from its producers to servers that are closer to its consumers. This dissemination reduces network traffic and balances load among servers by exploiting the geographic and temporal locality of reference exhibited in client access patterns. The level of dissemination depends on the relative popularity of documents, and on the expected reduction in traffic that results from such dissemination. Using extensive HTTP logs, the author and his colleagues devised an analytical model of server popularity and file access profiles. With that model, he shows that disseminating the most popular documents on servers closer to clients could reduce network traffic considerably, while balancing server loads. Trace driven simulations quantify the performance gains achievable through such a protocol.