Scalability challenges and solutions for emerging networks

Summary form only given. Computer networks are becoming increasingly common, and are used in sensitive applications in which serious damage could be done by a network failure. There is a need for design principles that would enable a new generation of solutions having the required properties. Needed are technologies that would be inherently robust, provably scalable, and sufficiently self-organizing to adapt as conditions change in the network. The Spinglass project has been successful in solving an important class of such problems. At the core of our work is a new style of gossip-based communication protocol. We are using this protocol in support of a variety of systems programming tools. The article discusses two of them: Bimodal Multicast, a scalable reliable multicast protocol having probabilistic reliability properties, and Astrolabe, a virtual distributed database constructed entirely through peer-to-peer interactions among the components of a large system. Both technologies are shown to be stable under stress, arbitrarily scalable without growth in communication or processing loads, and to have real-time properties.