IDPS Signature Classification with a Reject Option and the Incorporation of Expert Knowledge

As the importance of intrusion detection and prevention systems (IDPSs) increases, great costs are being incurred to manually manage signatures, which are malicious communication pattern files. Network security experts need to classify signatures by importance for an IDPS to work optimally. In this study, we propose and evaluate a machine learning signature classification model with a reject option (RO) to reduce the cost of setting up an IDPS. Experts classify some signatures with predefined if-then rules. We first design two types of features, symbolic features (SFs) and keyword features (KFs), which are used in keyword matching for the if-then rules. Next, we design web information and message features (WMFs) to capture the properties of signatures that do not match the if-then rules. The WMFs are extracted as the term frequency-inverse document frequency (TF-IDF) features of the message text in each signature and the text expanded by web scraping, respectively. Because failures need to be minimized when classifying IDPS signatures, we consider introducing an RO in our proposed model. The effectiveness of the proposed classification model is evaluated in experiments conducted on two real datasets composed of signatures labeled by experts. In the experiment, the SF and WMF combination outperforms the SF and KF combination. We also show that using a deep ensemble improves the performance of the RO. An analysis shows that experts refer to the text contained in the signatures and information from the web.