Performance comparison of visualization-based malware detection and classification techniques

Cybercriminals use malware or malicious software to cause harm to the victim. Malware is a continuous source of concern for security teams. Malware analysis techniques, including static, dynamic, hybrid, and memory analysis, are used to comprehend the behavior and its impact. The aforementioned malware analysis techniques require domain knowledge to extract the artifacts from suspicious files, which is not always possible. A visualization approach, in which malware files are transformed into images, is one of the recently used techniques by researchers for malware detection and classification. In this paper, we apply four widely used techniques based on the visualization using a new dataset of memory dump files of malware families and benign classes. These visualization techniques include a histogram of oriented gradients (HOG) with multilayer perceptron (MLP), convolutional neural network (CNN) with pretrained weight of visual geometry group 16 (VGG), Transfer learning of VGG16 with support vector machine (SVM), and integration of global image descriptor (GIST) and HOG with SVM. Among the selected techniques, CNN with a pretrained weight of VGG16 outperformed the other techniques in terms of accuracy, precision, recall, and f1-score. Apart from the performance metrics, the results of selected techniques are also analyzed in terms of computational cost and memory utilization.