An Efficient Single-pass Online Computation of Higher-Order Bivariate Statistics

Bivariate statistical moments are widely used in many areas of academic research and industry, beyond covariance matrices. Relationships between pairs of random variables can be further used to test hypotheses, which require higher-order bivariate moments. To reveal subtle patterns and differences, the processing of millions of observations across numerous variables may be necessary, posing significant challenges, especially for streaming data. This study introduces an efficient, single-pass methodology to compute central statistical co-moments of arbitrary order on large, continuous data streams. Our approach employs two key steps: (1) constructing a dense memory layout for input data batches enabling a single-pass matrix transformation, and (2) performing intermediate accumulation using optimized linear algebra subprograms (BLAS), resulting in a fast and scalable streaming solution. This method accelerates computation, essential for real-time analytics at scale. At any point, the accumulated data can be transformed to yield the desired higher-order bivariate moments. We also address numerical stability issues and propose a solution to maintain accuracy across millions of observations. Experimental results show that our method outperforms current streaming methods by more that an order of magnitude, advancing the extraction of bivariate statistical insights for data-intensive and time-sensitive applications.