Deep Learning for Malware Traffic Analysis: A Comprehensive Review on Model Optimization and Performance Enhancement

Deep learning-based malware detection has seen rapid progress over the past five years, in line with the increasing complexity of cyber threats and digital system vulnerabilities. This study presents a comprehensive review of the latest developments in applying deep learning for malware traffic analysis, focusing on optimization techniques and model performance improvement. Various approaches have been identified as key to improving accuracy and efficiency, including regularization techniques to reduce overfitting, hyperparameter optimization to enhance model generalization, learning rate scheduling to accelerate convergence, and selecting an optimizer suited to the characteristics of the malware dataset. From an architectural perspective, Transformers and Convolutional Neural Networks (CNNs) dominate the literature due to their ability to capture long-term dependencies and extract spatial features. Additionally, approaches such as Transfer Learning, Long Short-Term Memory (LSTM), Ensemble Learning, and Generative Adversarial Networks (GAN) further enhance model adaptability. Current trends indicate the increasing adoption of hybrid models and hierarchical attention mechanisms to address increasingly complex attack patterns. The main challenge ahead is to create models that are accurate but also transparent and resistant to advanced evasion techniques to support adaptive, real-time, and robust malware detection systems in AI-based cyber defense.