Enhancing System Security Monitoring with AI: Integrating the Anomaly Transformer and Prometheus for Real-Time Anomaly Detection

As cyber threats become increasingly sophisticated, traditional security monitoring methods often fall short. In particular, supervised learning models struggle to adapt to the ever-evolving nature of cyber-attacks, especially when dealing with large volumes of data and real-time defenses, as they cannot autonomously learn new attack patterns or operational contexts. This research proposes the use of an Anomaly Transformer (AT) in conjunction with Prometheus to detect abnormal insecure events in computer systems in real-time. Prometheus collects critical system metrics, including CPU, memory, and network usage, which the AT then analyzes to find anomalies. The study demonstrates that this AI-enhanced approach significantly improves threat detection accuracy and efficiency, achieving a high ROC area under the curve (AUC) of 0.82 and reducing false positives and negatives. Furthermore, the paper highlights the limitations of this approach in handling cyber-attacks that do not cause metric volatility and addresses the lack of real-world environment validation. The integration of these technologies represents a significant advancement in proactive cybersecurity, offering robust, scalable solutions to detect and mitigate complex threats in real-time.