Feature Extraction Techniques for Malicious System Calls

System-Call analysis is considered one of the most promising methods for malware detection due to its ability to identify new variants and zero-day threats. This research work focuses on developing a host intrusion detection system that detects malicious system calls using feature extraction techniques. Our goals are twofold: first, to analyze and compare existing feature extraction techniques for detecting these malicious calls, and second, to achieve high accuracy with low false-positive rates. In our study, we applied two main feature extraction methods, each divided into three groups, and used a Support Vector Machine (SVM) classifier to evaluate their performance. We measured the results using a confusion matrix, sensitivity, and fallout to make comparisons. To ensure fairness, all techniques were tested using the same classifier. Our findings show that both the n-gram and PCA (Principal Component Analysis) techniques are effective for feature extraction. Overall, n-grams outperformed all three variations of PCA, which used 20, 27, and 30 features. However, increasing the value of ’n’ led to a noticeable drop in accuracy. In contrast, PCA did not show a consistent relationship between the number of features and performance and generally performed worse than n-grams. Our proposed system achieved high accuracy and low false-positive rates. The highest accuracy scores were achieved by the 3-gram and 5-gram models, both scoring 0.96. The lowest false-positive rates (0) were obtained using PCA with 20 and 30 features.