Detecting Malicious Encrypted Traffic with Multimodal Representations

The rapid advancement of encryption technology enhances network security while enabling hidden attackers to avoid detection. Traditional methods for malicious encrypted traffic detection, which predominantly rely on a single modality such as statistical features or content representations, often fall short of adapting to dynamic network environments. Methods based on graph representations grapple with challenges such as insufficient modeling of the encryption properties and substantial computational resource requirements. Multimodal-based methods seldom consider the graph-based dynamic representation and often overlook the differences in feature spaces. Moreover, these methods are not evaluated for universality across platforms. To solve challenges above, we propose M2D, a multimodal-based framework for malicious encrypted traffic detection suitable for all versions of TLS protocols. M2D extracts (a) heterogeneous graph representation from spatial and temporal features to capture both dynamic patterns and complex interactions between different entities; (b) ciphertext visual representation to enhance content encapsulation; and (c) plaintext representation to explore semantics, then fuses them through the multi-head attention mechanism to emphasize more effective components. Furthermore, we set up an encrypted network traffic dataset generated by sandbox, with session keys embedded for decryption. Experimental results on both public and proposed datasets demonstrate the superior performance of M2D in binary and multi-class classification tasks. Additionally, ablation studies confirm the effectiveness of each component.