Scheduled System Maintenance:
Some services will be unavailable Sunday, March 29th through Monday, March 30th. We apologize for the inconvenience.
By Topic

On the benefits of a workflow-aware file system in high-performance computing systems

Sign In

Cookies must be enabled to login.After enabling cookies , please use refresh or reload or ctrl+f5 on the browser for the login options.

The purchase and pricing options are temporarily unavailable. Please try again later.
2 Author(s)
Yang Wang ; Dept. of Comput. Sci., Alberta Univ., Edmonton, Alta. ; Lu, P.

Traditional high-performance computing (HPC) systems have independent job schedulers and file systems that do not interact in substantial ways. We make the case that some integration of scheduler and file system can have three main benefits. First, the dataflow dependencies between the jobs in a workflow can be inferred by combining the scheduler's knowledge of the jobs (and possibly control-flow) and the file system's knowledge of the files accessed. Second, the dataflow information can be used to improve workflow instance concurrency when there are (potential) filename conflicts. Third, when workflows need to be re-computed, only the affected jobs need to be re-executed. We present the design and a simulation study of the Workflow-Aware File System (WaFS). Our design layers a namespace manager (NM) on top of existing file systems to provide, for example, a dataflow engine and a versioned file system. Our simulation study (with a specific set of application parameters) shows that a combined WaFS-aware file system and scheduler can significantly improve makespans for intensive workloads and be efficient in the re-computation of jobs

Published in:

High-Performance Computing in Asia-Pacific Region, 2005. Proceedings. Eighth International Conference on

Date of Conference:

1-1 July 2005