Hierarchical Deep Learning Approach for Malware Family Classification Using Resnext-50 Model

Malware categorization is very important for security aims because when known types of malicious software are distinguished, they can be effectively controlled and prevented. Signature based products do not work well because old methods cannot detect new ones or polymorphic Malware. Many other classifications like SVM and random forests are more accurate yet bring extra difficulty with feature engineering and are not scalable to classify large and complex datasets with multiple in number of classes. To overcome these challenges, deep learning techniques have been applied in this paper and the results of the convolutional neural network (CNN) in the classification of the malware images have been presented. Nevertheless, basic categories of CNNs including AlexNet and VGGNet tend to be computationally expensive and are very sensitive to overfitting when used in high dimensionality. To avoid the above limitations, this study presents the use of a deep residual network referred to as ResNeXt-50 with grouped convolutions for the multi-class malware categorization of $\mathbf{2 5}$ different types of malware. Due to the hierarchical feature extraction of ResNeXt-50, the representation of the patterns of malware types is efficient and accurate. The proposed model gets 98 % accuracy rate in classifying $\mathbf{1 5, 0 0 0}$ malware samples; it is clearly superior to the baseline methods. The contribution of hyperparameters like cardinality, plus data augmentation strategies is further illustrated through ablative analysis results. These findings point to the usefulness of ResNeXt-50 gives $\mathbf{9 8 \%}$ accuracy. It is a powerful foundation if used for malware classification that allows us to generalize results to a variety of domains and keep the computational complexity relatively low.