Building an Online Computing Service over Volunteer Grid Resources

Volunteer computing grids have traditionally been used for massively parallel workloads, such as processing data from large scientific experiments. We argue that the domain of volunteer grids can be extended well beyond this specific niche, by enhancing them with built-in mechanisms for integration with with standard clusters, grids and clouds, to compensate for unexpected fluctuations in resource availability and quality of service. The resulting capabilities for on-demand dynamic expansion of the resource pool, together with sophisticated scheduling mechanisms will turn volunteer grids into a powerful execution platform for on-line interactive computing services. We will show our experience with the GridBoT system, which implements these ideas. GridBoT is part of a production high performance online service for genetic linkage analysis, called Super link-online. The system enables anyone with the Internet access to submit genetic data, and easily and quickly analyze it as if using a supercomputer. The analyses are automatically parallelized and executed via GridBoT on over 45,000 non-dedicated machines from the Superlink@Technion volunteer grid, as well as on 9 other grids and clouds, including the Amazon EC2. Since 2009 the system has served more than 300 geneticists from leading research institutions worldwide, and executed over 6500 different real analysis runs, with about 10 million tasks consumed over 420 CPU years.