CREW: A Gossip-based Flash-Dissemination System

In this paper, we explore a new form of dissemination called Flash Dissemination that involves dissemination of fixed, rich information to a large number of recipients in as short a time as possible. Key characteristics of Flash Dissemination include unpredictability in its need, scalability to large number of recipients and autonomic performance in highly heterogenous and failureprone environments. Previous work either addresses large content delivery in heterogenous networks or fault-tolerant dissemination of (streaming) events. We investigate a peer-based approach using foundations from broadcast networks, gossip theory and random networks. In this paper, we propose CREW (Concurrent Random Expanding Walkers), a scalable, lightweight, and autonomic gossip-based protocol. CREW is also explicitly designed to maximize the speed of dissemination using adaptive and intelligent intra and inter node concurrency. We implemented CREW on top of a scalable middleware environment and compared it to optimized implementations of popular gossip and peer-based systems. Our experiments show that CREW outperforms both traditional gossip and current large content dissemination systems, across a wide range of comparative metrics, even though its design is counterintuitive from a systems perspective.