Machine learning to detect anomalies in web log analysis

As the information technology develops rapidly, Web servers are easily to be attacked because of their high value. Therefore, Web security has aroused great concern in both academia and industry. Anomaly detection plays a significant role in the field of Web security, and log messages recording detailed system runtime information has become an important data analysis object accordingly. Traditional log anomaly detection relies on programmers to manually inspect by keyword search and regular expression match. Although the programmers can use intrusion detection system to reduce their workload, yet the log system data is huge, attack types are various, and hacking skills are improving, which make the traditional detection not efficient enough. To improve the traditional detection technology, many of anomaly detection mechanisms have been proposed in recent years, especially the machine learning method. In this paper, an anomaly detection system for web log files has been proposed, which adopts a two-level machine learning algorithm. The decision tree model classifies normal and anomalous data sets. The normal data set is manually checked for the establishment of multiple HMMs. The experimental data comes from the real industrial environment where log files have been collected, which contain many true intrusion messages. After comparing with three types of machine learning algorithms used in anomaly detection, the experimental results on this data set suggest that this system achieves higher detection accuracy and can detect unknown anomaly data.