An Integrated Approach for Massive Sequential Data Processing in Civil Infrastructure Operation and Maintenance

This paper presents an extract-transform-load (ETL) approach based on multilayer task execution for processing massive sequential data collected from infrastructure operation and maintenance. The proposed approach consists of ETL task partition, execution mode selection, and ETL modeling. The task partition focuses on dividing the ETL process into four tasks to be executed in accordance with different organizational forms of data. Sequenced or non-sequenced load mode is optional, which is independent of the data standardization. In addition, the ETL modeling phase implements conceptual, logical, and physical modeling for the multi-dimensional model. Our main objective is to integrate massive sequential data, enhancing decision-making performance for the intelligent management platform. Traffic data for two years were collected from various systems and acquisition tools of different providers to evaluate the data integration capability of the proposed approach. Furthermore, Kettle software was used to perform transformation and job modules for the multilayer tasks. In addition, a machine learning algorithm was used to generate traffic warning in the tunnels based on the integrated data. The proposed approach is promising for the management and analysis of massive sequential data generated in operation, the maintenance of transportation tunnels, and effective decision-making.