TLS-Driven Reverse Shell Detection via Anomaly-Based Methods in Enterprise Network Traffic

Cybersecurity has become a prominent topic in enterprise behavior in Lithuania due to new regulatory standards (NIS2 directive) that will be incorporated into national law in Q4 of 2024. Furthermore, the boost in computerization has led to an increase in network traffic, particularly in the context of anomalies and intrusion detections. The reverse shell—a technique used by attackers in many exploits as a method of establishing access—has emerged as a significant intrusion tactic. Reverse shells are part of remote execution strategies outlined in the MITRE framework, as well as in resources such as the CVE database. In this paper, an enterprise dataset was collected, and several machine learning models, including eXtended Gradient Boosting (XGBoost) and Support Vector Machine (SVM), were evaluated along with a deep learning model in collaboration with an autoencoder. The dataset was split into two parts, representing TLS and TCP session features. Mutual Information (MI) Score was employed as the feature extraction method. The evaluation revealed that the most effective way to detect reverse shell exploits by examining encrypted traffic anomalies is through the use of XGBoost, achieving an F1-Score of up to 99.6% within the self-collected enterprise dataset containing 13,000 encrypted sessions. The future work will be to collect large scale dataset and test the accuracy within it.