Data Flow as Code: Managing Data Flow in an Industrial Hierarchical Edge Network

For real-time analytics and insight into an industrial production process, it is necessary to deploy a hierarchy of edge nodes that collect massive amounts of telemetry data from many field devices without violating data security constraints in the process core network. Depending on the analytics use case, the requirement for the data at an edge can vary from a hot data stream, warm aggregated data, and cold historical data. Data availability can be severely impacted when communication among edge nodes and devices is interrupted due to network failures. Data backfilling, along with current data transmission with low bandwidth in the process core network is the challenge. This paper proposes a concept for data management called Data Flow as Code (DFaC), which accommodates the requirements of different analytical applications and leverages the advantages of different methods, such as data aggregation, compression, and redundancy removal, and allows configuring edges based on data fingerprints (or data characteristics) to achieve maximum benefit with respect to bandwidth and the quality of the data. Also, we propose DataTrait, which shows a modular implementation of DFaC. We demonstrate the results of extensive numerical experiments conducted with an Azure hierarchical edge setup with open-source Pronto data and Azure predictive maintenance data. Only < 12% of the actual data needs to be backfilled when the connection is restored without compromising data quality, saving huge bandwidth and processing time.