TAPER: a two-step approach for all-strong-pairs correlation query in large databases
Xiong, H.
Shashi Shekhar
Tan, P.-M.
Vipin Kumar
Manage. Sci. & Inf. Syst. Dept., Rutgers Univ., Newark, NJ, USA;
This paper appears in: Knowledge and Data Engineering, IEEE Transactions on
Publication Date: April 2006
Volume: 18,
Issue: 4
On page(s): 493- 508
ISSN: 1041-4347
INSPEC Accession Number: 8944919
Digital Object Identifier: 10.1109/TKDE.2006.1599388
Current Version Published: 2006-02-27
Abstract
Given a user-specified minimum correlation threshold θ and a market-basket database with N items and T transactions, an all-strong-pairs correlation query finds all item pairs with correlations above the threshold θ. However, when the number of items and transactions are large, the computation cost of this query can be very high. The goal of this paper is to provide computationally efficient algorithms to answer the all-strong-pairs correlation query. Indeed, we identify an upper bound of Pearson's correlation coefficient for binary variables. This upper bound is not only much cheaper to compute than Pearson's correlation coefficient, but also exhibits special monotone properties which allow pruning of many item pairs even without computing their upper bounds. A two-step all-strong-pairs correlation query (TAPER) algorithm is proposed to exploit these properties in a filter-and-refine manner. Furthermore, we provide an algebraic cost model which shows that the computation savings from pruning is independent of or improves when the number of items is increased in data sets with Zipf-like or linear rank-support distributions. Experimental results from synthetic and real-world data sets exhibit similar trends and show that the TAPER algorithm can be an order of magnitude faster than brute-force alternatives. Finally, we demonstrate that the algorithmic ideas developed in the TAPER algorithm can be extended to efficiently compute negative correlation and uncentered Pearson's correlation coefficient.
Index
Terms
Available to subscribers and IEEE members.
References
Available to subscribers and IEEE members.
Citing Documents
Available to subscribers and IEEE members.
Cart
