Comparing unsupervised learning approaches to detect network intrusion using NetFlow data

Networks are vulnerable to costly attacks. Thus, the ability to detect these intrusions early on and minimize their impact is imperative to the financial security and reputation of an institution. There are two mainstream systems of intrusion detection (IDS), signature-based and anomaly-based IDS. Signature-based IDS identify intrusions by referencing a database of known identity, or signature, for each of the previous intrusion events. Anomaly-based IDS attempt to identify intrusions by referencing a baseline or learned patterns of normal behavior. Under this approach, deviations from the baseline are considered intrusions. We assume this type of behavior is rare and distinguishable from normal activity. Our research investigates unsupervised techniques for anomaly-based network intrusion detection. For this research, we use real-time traffic data from University of Virginia network. We evaluate the performance between Local Outlier Factor (LOF) and Isolation Forest (iForest) by probing the similarities and differences between the result of each approach. Distribution plots show there is a greater variation of attributes in anomalies identified by iForest than those anomalies identified by LOF. Furthermore, iForest results are more distinctive from all data than the LOF results. With the assumptions that anomalies are points that are rare and distinctive, we find that iForest performs well in identifying anomalies compared to LOF.