OnionGraph: A Graph Neural Network Framework for Tor-based Malware Traffic Detection and Classification

The Tor network, widely known for its anonymity, is frequently abused for cybercriminal activities. Many strains of malware use Tor to spread more covertly, making them harder to detect and trace. To counter this, we introduce OnionGraph, a malware detection and classification framework based on Graph Neural Networks (GNNs). We collected malicious traffic from recent Tor-based malware targeting Windows, extracted TLS (Transport Layer Security) sessions (bidirectional flows), and constructed a TLS session graph (TSG) to identify and classify Tor-based malware. By combining packet-level features with TSG structural features, OnionGraph uncovers richer traffic interaction patterns, avoiding the need for manual feature extraction. Additionally, we built an up-to-date Tor traffic dataset by combining malicious traffic with benign traffic and made it open-source, enabling us to evaluate OnionGraph's performance rigorously. Experimental results demonstrate that OnionGraph surpasses existing GNN-based methods and state-of-the-art Torbased malware detection techniques, achieving around 98% detection accuracy and a lower false negative rate (FNR). OnionGraph also consistently outperforms other machine learning and deep learning models in classifying Tor-based malware, achieving an accuracy of 97.8 %.