i-MagNet: A real-time intelligent framework for finding specific needles from needle stacks

Currently the volume of telecom network management data is expanding exponentially, mainly due to the explosive growth in the number of communicating devices along with the increase in heterogeneity of the networks. Such scale of data obsoletes the traditional approach of extracting offline analytics from the network traces governed by some pre-defined schemes. In order to increase the efficiency of the Operations Support System (OSS) and gain in-depth understanding of the generic relationship between network entities, the monitoring data needs to undergo large-scale deep analytics processing. In this paper we present i-MagNet, an integrated analytics framework developed with the popular real-time stream processing paradigm Storm. The components of i-MagNet intelligently micro-batch segments of incoming streams to enable high-throughput online analytics of management trace streams. Inter-dependence metrics (temporal and statistical) are exploited to extract contiguous event subsequences, which can then be independently examined as part of a network incident analysis system.