varMax: Towards Confidence-Based Zero-Day Attack Recognition

Detecting zero-day attacks, which exploit unknown vulnerabilities, is vital in mission-critical systems. Deep Neural Networks (DNNs) often fails to identify unknown activity, as they make overly confident predictions due to SoftMax function, effective at identifying known attacks but is not structured to identify unknown activity patterns. Open-Set recognition (OSR) algorithms designed for DNNs tend to flag inputs as unknown, needing a balanced approach. To address this, we introduce varMax, a bias-neutral OSR technique using DNN logit variance to distinguish known from unknown inputs. It has three components: (1) a top-difference algorithm comparing top two softmax scores to a threshold, (2) a method classifying ambiguous samples based on logit variance, and (3) an energy-based out-of-distribution function enhancing classification accuracy and trustworthiness. Our evaluation shows varMax outperforms leading methods in identifying unknown activities and improves DNN confidence and robustness in distinguishing between known and unknown inputs. This research marks a significant step forward in the development of reliable and unbiased intrusion detection systems for cybersecurity threats.