Measuring the Complexity of Benchmark NIDS Datasets via Spectral Analysis

Due to the difficulties in obtaining real-world network traffic, lab-based datasets are commonplace in ML-based NIDS research. However, despite their popularity, whether these datasets conform to the expected properties of real-world traffic is unclear. To elucidate this, we present a novel measure of input complexity based on spectral clustering analysis. Via this measure, we evaluate two popular benchmark NIDS datasets, contrasting their input complexity with both real-world network traffic and benchmark image datasets. We find that the volumetric attack classes contain minimal data diversity, potentially limiting researchers' abilities to evaluate their methodologies. Following this, we produce our own attack traffic with considerably higher input complexity, demonstrating that, despite their repetitive nature, low data diversity is not a universal property of volumetric attacks. Our work highlights likely limitations of synthetic benchmark datasets for evaluating machine-learning based NIDS as well as the importance of input complexity in network intrusion dataset design.