REDA: Malicious Traffic Detection Based on Record Length and Frequency Domain Analysis

The TLS encryption protocol plays a vital role in securing data transmission, but it also presents challenges for payload-based Network Intrusion Detection Systems (NIDS). Existing methods utilize statistical characteristics of side-channel features, such as the mean packet length, to identify encrypted traffic. However, packet lengths are constrained by the Maximum Segment Size (MSS) of the TCP protocol. This constraint causes the length-varied sequence to be encapsulated into segments of equal length, resulting in information loss. Moreover, flow-level statistical features are vulnerable to interference from noisy packets, making it challenging to detect malicious traffic injected with benign packets effectively. In this paper, we propose an encrypted traffic detection model based on Record length and frEquency Domain Analysis (REDA). First, we reconstruct the TLS Record Length Sequence (TRLS), which is a length-varied sequence, to capture differences in traffic content during transmission. Second, we employ the Discrete Fourier Transform (DFT) to extract frequency domain features of the TRLS, which are resistant to attacker interference. Finally, an improved One-Class Support Vector Machine (OCSVM) algorithm is devised for the unsupervised detection of malicious traffic, enabling the identification of unknown attacks. Experiments show that REDA is superior to other state-of-the-art methods in terms of accuracy by 2.44%.