Grid Appliance — On the design of self-organizing, decentralized grids

“Give a man a fish, feed him for a day. Teach a man to fish, feed him for a lifetime” - Lau Tzu Grid computing projects such as TeraGrid, Grid'5000, and OpenScience Grid provide researchers access to vast amounts of compute resources, but in doing so, require the adaption of their workloads to the environments provided by these systems. Researchers do not have many alternatives as creating these types of systems involve coordination of distributed systems and expertise in networking, operating systems, security, and grid middleware. This results in many research groups creating small, in-house compute clusters where scheduling is often ad-hoc, thus limiting effective resource utilization. To address these challenges we present the “Grid Appliance.” The “Grid Appliance” enables researchers to seamlessly deploy, extend, and share their systems both locally and across network domains for both small and large scale computing grids. This paper details the design of the Grid Appliance and reports on experiences and lessons learned over four years of development and deployment involving wide-area grids.