Malware Detection With Subspace Learning-Based One-Class Classification

Detecting malware is crucial for ensuring the security of computer systems. Traditional machine learning models face challenges in effectively detecting malware, mainly due to the class imbalance problem, where the number of malware samples is significantly smaller than that of non-malware samples. Additionally, malware’s dynamic and evolving nature, continuously altering its structure and tactics, presents a substantial challenge for conventional artificial intelligence algorithms, further complicating the detection task. In pursuing an optimized malware detection technique, researchers initially explored traditional machine learning algorithms, focusing on the features of Portable Executable (PE) file headers. However, the inherent issues, such as imbalanced datasets and the deceptive nature of malware, have raised concerns about the credibility of the attained results. This can result in misclassifying malware as non-malware, leading to security vulnerabilities. One-Class Classification (OCC) methods have emerged as a promising approach to improve the detection of unknown malware. However, traditional OCC approaches face the challenge of the curse of dimensionality. This research proposes adapting subspace learning-based OCC methods to overcome the curse of dimensionality and effectively handle the class imbalance problem. We propose a pipeline for detecting malware using methods that jointly optimize a subspace and data description for OCC. We evaluate the performance of various one-class classifiers on three different datasets. We observed that the subspace-learning-based OCC is a promising approach. Evaluating various classifiers on three datasets reveals promising results, with a True Positive Rate (TPR) of 100 % for subspace-learning-based OCC. The proposed pipeline can serve as a valuable tool for improving the security of computer systems by accurately detecting malware and protecting against potential attacks.