Anomaly Detection in Edge Streams Using Term Frequency-Inverse Graph Frequency (TF-IGF) Concept

The ability to detect fraudulent activities such as denial-of-service attacks (DoS), social media impersonations, or fake recommendations in e-commerce networks is a difficult task. One approach to aiding with detection of these types of anomalous activities is through representing the data as a graph. However, the scalability of graph-based approaches is problematic. In general, given a stream of graph edges, where each edge indicates a communication/link in a certain time, the objective is to detect anomalous edges that represent actions such as DoS attacks, port scans, etc., and do so in an online manner while consuming constant time and memory. In this paper, we propose Term Frequency-Inverse Graph Frequency (TF-IGF) as an online anomaly detection approach, which assigns anomaly scores to edge streams. TF-IGF offers the following contributions: (1) streaming; processes one edge at a time, (2) online; detects suddenly appearing anomalous edges in near real-time, (3) accurate; has better accuracy than the state-of-the-art approaches, and (4) efficient; consumes constant time and memory to process edges. We evaluate this approach on both synthetic and real-world data sets.