Outlier-based Anomaly Detection in Firewall Logs

Nowadays, most corporations or government agencies adopt various cybersecurity detection and protection systems to safeguard their assets. Currently available cybersecurity products include firewalls, intrusion detection systems (IDS), intrusion prevention systems (IPS), web application firewalls (WAF), anti-virus software, distributed denial-of-service (DDoS) protection systems, and advanced persistent threat (APT) protection systems, among many others. This article proposes the use of unsupervised learning methods to detect anomalous behavior in network environments. In order to find critical behavioral features, we attempted to extract and analyze different fields, and ultimately discovering two key behavioral features: (1) the number of daily connections from a source IP address to different external hosts, and (2) the total number of daily connections from a source IP address to a destination IP address. This method simplifies the complexity of various cybersecurity devices and introduces our designed long-cycle analysis method. It also solves the problem of not being able to define baseline behavior due to the high cost involved. Through unsupervised learning and feature compression, our proposed methodology allows monitoring targets to self-compare and self-verify their own behavior. Our proposed methodology can figure out anomalous behavior in over 170 customers, billions of source IP addresses, and trillions of network connections.