Understanding Data Pre-processing with a Hybrid Diagram Integrating ER and Data Flow Notation

In a data science project, a significant effort is spent pre-processing databases, text and image data sets, before a machine learning model can be computed. Moreover, the analytic process is iterative, requiring a team of data scientists to add and remove features (attributes) from the target data set going all the way back to data sources. However, such programming effort is carried out without a data model behind, which results in redundant and inconsistent data sets and source code difficult to extend and maintain. On the other hand, the ER model has a proven track record to design databases before developing source code. Heeding the importance of data-centric models to store and analyze data, we propose a hybrid diagram (FLOWER=FLOW+ER) mixing data objects and processing flow, under the CRISP standard. Specifically, our novel diagram mixes ER entities (data objects) and processing steps (function calls). We explain how diagram creation can be partially automated by parsing source code. We present two case studies illustrating how our proposed hybrid diagram works in two challenging problems from biomedical engineering and computer vision: (1) detecting and classifying spikes in biomedical signals and (2) labeling identified objects in images. We argue our diagram can reduce code development time, enhance team collaboration and eliminate redundant data. We hope our work will motivate research bridging database design, project management and machine learning.