Cluster Computing on the Fly: resource discovery in a cycle sharing peer-to-peer system

Peer-to-peer computing, the harnessing of idle compute cycles throughout the Internet, offers exciting new research challenges in the converging domains of networking and distributed computing. Our system, Cluster Computing on the Fly, seeks to harvest cycles from ordinary users in an open access, non-institutional environment. We conduct a comprehensive study of generic searching methods in a highly dynamic peer-to-peer environment for locating idle cycles for workpile applications which are heavy consumers of cycles. We compare four scalable search methods: expanding ring, advertisement-based, random walk and rendezvous point. We model a variety of workloads, simple scheduling strategies and stabilities of hosts. Our preliminary results show that under light workloads, rendezvous point performs best, while under heavy workloads, its performance falls below the other techniques. We expected rendezvous point to consistently outperform the other search techniques because of its inherent advantage in gathering knowledge about the idle cycles. However in a peer-to-peer environment, which satisfies requests on-demand, large jobs may dominate, resulting in delays for scheduling smaller jobs.