Hierarchical Probabilistic Feature Extraction for Optimized Feature Selection in Intrusion Detection Systems

Intrusion Detection Systems (IDS) requires the effective feature selection to enhance accuracy and efficiency, especially when handling the extensive datasets. This paper introduces a novel algorithm, Hierarchical Probabilistic Feature Extraction (HPFE) designed to optimize the feature selection for IDS applications using NSL-KDD dataset. HPFE integrates the hierarchical clustering with probabilistic weighting to systematically identify and prioritize the relevant features, reducing redundancy and improving classification accuracy. Initially, HPFE organizes the features into clusters based on statistical similarity, forming a hierarchical structure that highlights the distinct feature groups. Within each cluster, a probabilistic weight is assigned to each feature which is calculated using a combined metric of information gain and variance, capturing its significance in intrusion detection. An iterative feature pruning process, coupled with threshold adjustments and a feedback loop from model-based feature importance progressively refines the feature subset by maximizing relevance and minimizing computational requirement. Experimental results demonstrate HPFE’s effectiveness in accurately identifying the critical features while reducing the computational load, making it highly suitable for the NSL-KDD dataset. By addressing the limitations in traditional feature selection methods, HPFE provides a scalable and efficient solution for network security applications in IDS with the potential for adapting in the other data-intensive security domains.