SciPipe—Turning Scientific Workflows into Computer Programs

Scientific workflows are becoming increasingly popular as a way to automate complex scientific computations consisting of multiple programs. One of the main motivations behind this development is increased robustness and reproducibility of computational analyses. Chaining together multiple programs using plain scripts, as is often the first step in automating a pipeline, can easily become fragile and error prone due to the manual management of file paths and program invocations. Also, plain scripts are not optimal if for some reason you have to cancel a run and try to restart it from any partially finished steps. It can be hard to know which output files are properly finished and which are truncated from the cancelled run. Last but not least, plain scripts do not by default save an execution trace of what was run, such that the full procedure used to create a specific output file can be clearly presented. These are all aspects that scientific workflows are designed to help with.