Load management for scaling up Internet services

As the global Internet traffic increases, many popular sites are often unable to serve their TCP/IP workload, particularly during peak periods of activity. For example, Web servers for sports events are often swamped by requests during and after games. To address this problem, many sites allocate multiple server hosts to concurrently handle the incoming requests. To support workload sharing, they need a method to distribute the requests among the servers. Since network traffic is self-similar, with waves of heavy traffic at peak times, this requires dynamic feedback control. In this presentation we analyse several solutions to this scaling problem (client-based and DNS-based), and show some of their deficiencies. We then present our preferred method, which is based on IP packet forwarding and is transparent to both clients and servers. We implemented a TCP/IP load-management tool, NetDispatcher, that enables scalable, heterogeneous TCP/IP server clusters that can handle millions of TCP connections per hour. NetDispatcher does not perform any TCP/IP header translations. Hence, outgoing server-to-client packets need no processing and can follow a separate network route to the clients, resulting in improved bandwidth utilization and lower latency. Netdispatcher transparently handles server failures. NetDispatcher failures are handled by a shadow node without loosing active connection.