Resource Selection in Large-Scale Distributed System Using Dynamic Task Sharing

Large-scale distributed system provides an attractive scalable infrastructure for network applications. In such environment there exist large sets of heterogeneous and geographically distributed resources. These resources can be aggregated as a virtual computing platform for executing large-scale scientific applications. Among numerous optional resources, selecting appropriate resources for applications is challenging and affected by many factors. The loosely coupled nature of large-scale distributed environment makes data access unpredictable and instability. Slow allocation process may offset the benefit obtained by running a job on a fast node. Besides, the operation condition of a resource provider changes rapidly. The status of job execution and computing capability of a resource provider need to be considered dynamically. In this paper we present an approach of dynamic task-sharing based on the record of previous data download and current execution status of resource providers to select the appropriate one or more providers to execute a job together. The proposed approach can also avoids single point failure and server overload problem.