A hierarchical based ensemble classifier for behavioral malware detection using machine learning

Malwares are increasingly threatening the security and confidentiality of data. Therefore, the issues related to malware detection are gaining interest among the researchers. In this research work, an efficient behavioral malware detection system has been proposed for Portable Executable (PE) files. The detection of malware is done through machine learning classifiers. The most recently published dataset (containing samples from august 2019 to September 2020) namely, Blue Hexagon Open Dataset for Malware Analysis (BODMAS) has been used to train and test the proposed design. The proposed methodology is divided into two stages. First stage contains a binary classifier which detects whether PE file is malicious. A random forest is used as a binary classifier for this stage. Second stage contains a multi-class ensemble base voting classifier, and it detects the family of malware. K-nearest neighbor (KNN), support vector machine (SVM), random forest, decision tree and gradient boosting are used in voting classifier with equal weights. The proposed methodology achieved significant results with 99.48% accuracy in the first stage (binary classifier) and 92.49% accuracy in the second stage (ensemble-based classifier) on BODMAS dataset.