Proposing an efficient approach for malware clustering

Recently, malwares in security threats have a top rank which can damage computing systems and networks seriously. Over time malwares become more complicated and detection of them gets harder. Because traditional techniques such as signature based were not successful to detect metamorphic malwares, machine learning algorithms have been used to detect them. The Hidden Markov Model (HMM) has been successfully used in speech recognition, pattern recognition, part-of-speech tagging and biological sequence analysis. Previous work has shown that HMM is a convincing method for malware detection. However, some advanced metamorphic malwares have demonstrated to be more challenging to detect with these techniques. In this paper, we use clustering techniques with the probabilities as features based on HMM to the malware detection problem. In fact, we use clustering as classifier to detect malware. We compute clusters with K-means and Expectation Maximization algorithms. Results revealed that using clustering instead of HMM based approach, can have reasonable accuracy for metamorphic malware detection.