Distributed Mining and Modeling of Dynamic Lead-Lag Relations in Evolving Entities

Discovering and modeling lead-lag relations is a critical task in a variety of domains, including energy management, financial markets and environment monitoring. This task becomes more challenging when processing massive and highly dynamic data sources, often produced by sensors and live feeds that collect data about evolving entities in the real world. To cope with this data volume and velocity, distributed real-time computation systems have been proposed in the last years, although the problem of the lead-lag relation mining and modeling has not been deeply explored in this context. In this paper, we propose DL2-Miner, a novel distributed data mining framework for lead-lag relations based on this computational paradigm. DL2-Miner addresses the fundamental data mining task of uncovering interactions in evolving entities, and encompasses a lead-lag relation detection module with communication and computation optimization and a probabilistic model for lead-lag relation occurrence inference. It is implemented on top of the open source distributed real-time computation system Apache Storm, and preliminary experiments show promising results of our approach.