A Novel Approach to Detect Malware Variants Based on Classified Behaviors

An application programming interface (API) is an excellent feature since it is a procedure call interface to an operating system resource. Behavior features based on API play an important role in analyzing malware variants. However, the existing malware detection approaches have a lot of complex operations on construction and matching. Graph matching is an NP-complete problem and is time-consuming because of computational complexity. To address these issues, a promising approach is proposed to construct the classified behavior features from different malware families. In the proposed approach, a classified behavior feature consists of a kernel object (an API call parameter) and a series of operations (an API trace). Besides, a classified behavior graph (CBG) is represented as a number by hash to reduce workload and matching time. Subsequently, multiple machine learning classifiers are used for system classification. In particular, to verify the efficiency of our approach, we perform a series of experiments with different families. The experiments on 1220 malware samples show that the true positive rate is up to 88.3% and the false positive rate keeps within 3.9% by the support vector machine (SVM).