Hybrid Anomaly Detection Architecture for IDS/IPS Leveraging Ensemble and Deep Learning Models

The identification of zero-day exploits, disguised payloads, and polymorphic malware presents a significant challenge for conventional intrusion detection and prevention systems (IDS/IPS), leading to missed detections or excessive false positives. This research provides a robust hybrid anomaly detection architecture that takes a shot at the problem by merging signature-based and anomaly-based approaches with models from ensemble learning and deep learning. Specifically, XGBoost and Random Forest are applied for high-precision classification of known intrusions, while Autoencoders, Isolation Forest, and Transformer-based models capture novel attack patterns and temporal dependencies in network traffic. Experimental evaluation on benchmark datasets CICIDS2017 and UNSW-NB15 following normalization, encoding, and feature selection demonstrates the effectiveness of the hybrid framework. The Transformer-based models attained a ROCAUC of $\mathbf{9 6. 8 \%}$, while the Autoencoders and Isolation Forest models attained a rate of anomaly detection of 95.4%. Total accuracy was 98.1%, precision was 98.2%, recall was 98.3%, and F1-score was $\mathbf{9 8. 3 \%}$. These results demonstrate a significant improvement over standalone models. These results confirm that the proposed architecture reduces false positives, enhances detection reliability, and is suitable for real-time deployment in modern cybersecurity environments.