A Robust Anomaly Detection Framework Utilizing Feature Engineering and Ensemble Techniques with BETH Data

Anomaly detection holds a critical role in cybersecurity, with the aim of identifying suspicious activities or behaviors that deviated from normal patterns. In this paper, the BPF-extended tracking honeypot (BETH) dataset is utilized, designed specifically for uncertainty and robustness benchmarking in cybersecurity. BETH consists of essential features and the target variable evil. This dataset captured modern host activity and data collected from cloud services, rendering it highly relevant for real-world deployments. Notably, BETH offered a comprehensive view of malicious behavior by incorporating both kernel-process and network logs. The study aimed to apply various machine learning algorithms, encompassing Adaboost, XGBoost, Gradient Boosting, Random Forest, and Extra Trees Classifier, to detect anomalies in the BETH dataset. Prior to model training, preprocessing techniques were employed to handle missing values, perform feature engineering, and scale numerical features. Additionally, the performance of these algorithms was compared, and exploration into ensemble techniques stacking and voting classifiers to enhance detection accuracy. Through systematic experimentation and evaluation, the goal was to identify the most effective anomaly detection algorithms and ensemble strategies for enhancing cybersecurity measures.