By Topic

A Self-Optimizing Computation Partitioning Algorithm for Distributed Many-Task Computing

Sign In

Cookies must be enabled to login.After enabling cookies , please use refresh or reload or ctrl+f5 on the browser for the login options.

Formats Non-Member Member
$31 $13
Learn how you can qualify for the best price for this item!
Become an IEEE Member or Subscribe to
IEEE Xplore for exclusive pricing!
close button

puzzle piece

IEEE membership options for an individual and IEEE Xplore subscriptions for an organization offer the most affordable access to essential journal articles, conference papers, standards, eBooks, and eLearning courses.

Learn more about:

IEEE membership

IEEE Xplore subscriptions

5 Author(s)
Huashan Yu ; Sch. of Comput. Sci. & Electron. Eng., Peking Univ., Beijing, China ; Yingnan Li ; Xianguo Wu ; Jian Xiao
more authors

Many-task computing (MTC) is a practical paradigm for developing loosely coupled and complex scientific applications. In this paradigm, computation on a large dataset is decomposed into tasks that are expected to be executed in parallel with dynamically allocated computing resources. These tasks pass data via files, and each one is to execute an existing program on one dataset element. Task scheduling is a key issue to enable MTC on parallel platforms like large-scale clusters, Grids and Clouds. Current solutions mainly focus on maximizing the number of utilized parallel computing resources. This paper proposes a configurable MTC model that aims to minimize a MTC computation's turnaround time cost with as few resources as possible. The primary strategy is to coalesce tasks with application-specific expertise into task-sequences, and assign tasks on granularity of task-sequences. Based on this model, a self-optimizing task partitioning algorithm has been devised for scheduling tasks in MTC. It separates task assignment from resource allocation, and makes a tradeoff between maximizing utilized resources, balancing workload and reducing computation-scheduling overhead. The algorithm has been implemented in Harmonia, which is a software platform developed by Peking University for enabling MTC on large-scale distributed platforms. Both the configurable MTC model and the self-optimizing task partitioning algorithm were evaluated with the genome alternative splicing application, and experimental results have proved the model's practicability.

Published in:

ChinaGrid Conference (ChinaGrid), 2010 Fifth Annual

Date of Conference:

16-18 July 2010