n-Cycle: a set of algorithms for task distribution on a commodity grid

The global Internet is rich in commodity resources but scarce in specialized resources. We argue that a grid framework can achieve better performance if it separates management of commodity tasks from the management of the tasks requiring specialized resources. Assuming a relative homogeneity of the commodity resource providers, the determining factor of grid performance becomes the latency of entering into execution. This effectively transforms the resource allocation problem into a routing problem. We present an approach in which commodity tasks are distributed to the commodity service providers by request forwarding on the n-cycle overlay network. We provide algorithms for task allocation and for the maintenance of the overlay network. By ensuring that the algorithms use only narrow local information, the approach is easily scalable to millions of nodes. For task allocation algorithms in a commercial setting, fairness is of paramount importance. We investigate the properties of the proposed algorithms from the fairness point of view and show how adding several hops of random pre-walk to the algorithm can improve its fairness. Extensive simulations prove that the approach provides efficient task allocation on networks loaded up to 95% of their capacity.