TCMal: A Hybrid Deep Learning Model for Encrypted Malicious Traffic Classification

Encryption protocols can protect personal privacy, but attackers can also use it to evade detection by intrusion detection systems. If an intrusion detection system can identify the malware family class to which the encrypted malicious traffic belongs, it can take targeted measures to mitigate the damage. Traditional payload-based methods are no longer effective for encrypted malicious traffic classification, while deep learning-based methods have made some progress. In this paper, we propose a hybrid deep learning model TCMal, which consists of two sub-networks: T-net and C-net. T-net can be unsupervised pre-trained through transformer for representation learning of raw packets, while C-net for flow-level feature extraction through convolutional neural network. The experimental results show that TCMal outperforms the other four models and a control group model on all three datasets(the average F1-Score is 91.71%, 95.56%, and 91.92% on the three datasets, 2.8%, 3.51%, and 5.66% higher than the second place model, respectively). TCMal also proves the feasibility of transformer in encrypted malicious traffic classification.