An improved clustering validity index for determining the number of malware clusters

Nowadays, along with the development of the malware writing techniques, the diversity and amount of malware variants are constantly increasing and proliferation of these malware has posed major threats to the network computer users. However, variants of malware families always share typical characteristics which reflect their origin and purpose. Therefore, categorizing malware to different families is one of the computer security topics that are of great interest. A fundamental and difficult problem for malware clustering is the determination of the “true” number of malware families on real data collection. To address this issue, in this paper, resting on the analysis of the extracted instruction of malware samples, we propose an improved clustering validity index VNFS for determining the number of malware clusters based on hierarchical clustering algorithm for malware categorization. V
NFS
is defined based on the concepts of the compactness within each cluster and distances of the clusters representatives. By using a novel function proposed in this paper for total separation between clusters, V
NFS
can deal with different density and irregular data sets more robustly. A comprehensive experimental study on real daily collections of PE malware samples is performed to compare various clustering validation measure methods, such as RS-RMSSTD, PST2, V
FS
and V
XB
. Promising experimental results demonstrate that our proposed clustering validity index VNFS always leads to better cluster number than other clustering validity indices for hierarchical clustering and the number of malware clusters it recommends is almost the same as the virus analysts suggest.