A Technique for Efficient Query Estimation over Distributed Data Streams

Distributed data stream mining in a sliding window has emerged recently, due to its applications in many domains including large Telecoms and Internet Service Providers, financial tickers, ATM and credit card operations in banks and transactions in retail chains. Many of these large-scale applications prohibit monitoring data centrally at a single location due to their massive volume of the data; therefore, data acquisition, processing, and mining tasks are often distributed to a number of processing nodes, which monitor their local streams and exchange only the summary of data either periodically or on demand. While this offer many advantages, distributed stream applications possess significant challenges including problems related to an online analysis of the recent data, communication efficiency and various estimation of various complex queries. There are few existing techniques which solve problems related to distributed sliding window data stream; however, those techniques are focused on solving only simple problems and require high space, query, and communication cost, which can be a bottleneck for many of these large scale applications. In this paper, we propose an efficient query estimation technique by constructing a small sketch of the data stream. The constructed sketch uses a deterministic sliding window model and can estimate various complex queries, for both centralized and distributed applications; including point queries (i.e., range queries and heavy hitter queries), quantiles, inner product, and self-join size queries, with deterministic guarantees on the precision. The proposed approach improves upon recent existing work for these problems, in terms of the memory and query cost in a centralized setting and in terms of communication cost and merge complexity in a distributed setting. It requires O(1/ε
²
1 log (εN)) memory (where 0 <; ε <; 1 is a user defined parameter), can provide estimates in O(1) time, and processes each incoming record in O(1) amortized time. Detailed experimental analysis, both in centralized and distributed settings demonstrates that in practice the proposed approach uses about six times less memory, and has about eight times less query time when compared to ECM sketches. In a distributed application, the proposed technique also significantly improves (around seven times) on the communication cost between distributed sites.