A Multi-feature Fusion Method for Web Scanning Behavior Detection in Online Web Logs

Web scanning is generally a precursor of a malicious cyber-attack chain to discover system security vulnerabilities. The success or failure of a web attack is generally dependent on the results from the scanning process. Detecting web scanning behaviors is crucial for preventing malicious attacks. This paper proposes a Multi-Feature Fusion approach (MFF) that constructs multiple lightweight AI classifiers to detect scanning behaviors through web traffic logs. This model considers HTTP textual content, status codes and request frequency extracted from logs as main features to identify scanning attacks. We evaluated our MFF approach on 1.49 million real traffic logs collected by a company’s cyber security platform and achieved an average accuracy of 97.89%, showing better detection performance than the original WAF results. We also implemented the model in the company’s actual production environment to detect web scanning behaviors in real-time. The model ran stably with an average accuracy of 94.56% and a low FNR (False Negative Rate) of 2.62%. The evaluation results prove the effectiveness of our method for detecting web scanning behaviors in real environment.