Efficient Query Method for Collaborative Sharing of Monitoring-Based Data Streams Based on a Distributed Platform

Currently, an increasing number of high-frequency monitoring data streams are generated through the Internet of Things and various sensors. Window-based query calculation of multiple continuous data streams is inefficient because there are several repeated calculations between different query windows. An efficient query method for collaborative sharing of monitoring-type data streams to improve the data computation efficiency was proposed in this study, which was based on a distributed processing platform. The calculation results could be shared between different query windows by this method, which avoided several non-essential calculations. Multiple shared windows were generated by the calculation results of the query windows that could be subscribed by subsequent windows. Different from previous research, the same aggregation operation was not performed for all windows in this study. The execution plan was completed by constructing sequential processing windows. Each computation result of windows was composed of a series of intermediate or final aggregate results from the processed windows. Meanwhile, intermediate and final results of the windows were simultaneously stored and declared, in order to be used by the next window that is about to be processed. Results demonstrate that the collaborative shared query algorithm can save more than 90% of CPU resources and improve the computational efficiency of the data streams, compared with the TriWeave algorithm which is the most typical approach currently, as the number of queries increases. The proposed method provides a reference for sharing the results of multiple consecutive queries to improve computational efficiency in the process of data stream querying, analysis, and mining.