DEF-PIPE: Domain Specific Language Visualization for Big Data Pipelines

The complexity of Big Data analysis requires a combination of different software components into a pipeline performing different analysis steps. Supporting such pipelines requires different expertise provided by different actors: domain experts and technical/computing experts. The main objective of this work is to support domain experts with a tool for pipeline description, which does not require deep technical knowledge about the deployment and execution of Big Data pipelines. We present a solution to visualize Big Data pipeline description using the DEF-PIPE tool. The solution shows that the process of pipeline description is simple and intuitive for users who are not experts in computing. At the same time, DEF-PIPE automatically generates a textual description of the designed data pipelines, which contains the necessary information for simulation, adaptation, deployment, and resource management of the pipeline. In this case, a separation of concerns between the design and run-time phases of the Big Data pipeline lifecycle is supported. This solution allows us to bridge the gap between domain and technical experts. Providing libraries of steps and pipelines also allows the reusing of previously developed solutions.