SciFlow: A Composable Framework for Developing Scientific Workflows on HPC Clusters

Scientific workflows are widely used for simulation, analysis and prediction in the scientific domain. Coordination between components comes under scrutiny, owing to the complex interactions necessary to construct scientific workflows. These interactions are further complicated in dynamic workflows, where the path of the workflow needs to be adjusted during execution, based on outcomes at significant decision points. This paper presents a coordination paradigm for High Performance Computing (HPC), enabling scientists to conveniently express complex component connections. Channels, which provide architectural expressiveness, are used as the primitives to compose these complex connectors. Separation of control and computation is highlighted since most modules used in scientific workflows are reusable, trusted components. SciFlow exploits the computational power of HPC environments to parallelize execution and enables users to conveniently manipulate workflows using a range of connectors, thereby orchestrating execution.