Enhancing Big Data Security Through Anomaly Detection

Securing the massive and fast-moving data streams typical in Big Data environments presents unique challenges that traditional static security measures simply can't handle. To effectively protect these data flows, we need methods that can analyze traffic in real-time and respond swiftly to potential threats. Anomaly detection is one such method, offering an automated way to identify unusual or suspicious activities within Big Data systems. In this study, we explore several widely-used anomaly detection algorithms, evaluating their effectiveness in identifying anomalies within large datasets. Specifically, we will assess these algorithms using the UNSW-NB15 Dataset, aiming to pinpoint which algorithm, or combination of algorithms, is best suited for the demands of Big Data security.