By Topic

Improving Scientific Workflow Performance Using Policy Based Data Placement

Sign In

Cookies must be enabled to login.After enabling cookies , please use refresh or reload or ctrl+f5 on the browser for the login options.

Formats Non-Member Member
$31 $13
Learn how you can qualify for the best price for this item!
Become an IEEE Member or Subscribe to
IEEE Xplore for exclusive pricing!
close button

puzzle piece

IEEE membership options for an individual and IEEE Xplore subscriptions for an organization offer the most affordable access to essential journal articles, conference papers, standards, eBooks, and eLearning courses.

Learn more about:

IEEE membership

IEEE Xplore subscriptions

3 Author(s)
Amer, M.A. ; Univ. of Southern California, Los Angeles, CA, USA ; Chervenak, A. ; Weiwei Chen

I/O intensive jobs such as stage-in, stage-out or data clean-up jobs account for significant time in execution of scientific workflows. Workflow managers typically add these data management operations as supporting jobs to computational tasks with scheduling emphasis on compute jobs only. We present the integration of the Pegasus Workflow Management System with a Policy Based Data Placement Service (PDPS) to reduce overall workflow execution time. Pegasus delegates all data staging jobs to PDPS, which schedules and executes stage-in jobs based on selected data placement policies and simply executes stage-out and clean-up jobs independent of the workflow execution state. We measure the impact of using PDPS with Pegasus first with the Montage workflow, and then with a synthetic workflow. We enforce two policies and demonstrate the advantage of using PDPS for asynchronous data placement for scientific workflows. Our results show that the influence of PDPS on the overall workflow runtimes is dependent on the data characteristics of the executable workflow and the data placement policy being enforced.

Published in:

Policies for Distributed Systems and Networks (POLICY), 2012 IEEE International Symposium on

Date of Conference:

16-18 July 2012