A probabilistic model to preprocess darknet data for cyber threat intelligence generation

Internet traffic destined to routable yet unallocated IP addresses is commonly referred to as telescope or darknet data. Such unsolicited traffic is frequently, abundantly and effectively exploited to generate various cyber threat intelligence related, but not limited to, scanning activities, distributed denial of service attacks and malware identification. However, such data typically contains a significant amount of misconfiguration traffic caused by network/routing or hardware/software faults. The latter not only immensely affects the purity of darknet data, which hinders the accuracy of inference algorithms that operate on such data, but also wastes valuable storage resources. This paper proposes a probabilistic model to preprocess darknet data in order to prepare it for effective use. The aim is to fingerprint darknet misconfiguration traffic and subsequently filter it out. The model is advantageous as it does not rely on arbitrary cut-off thresholds, provide separate likelihood models to distinguish between miscon-figuration and other darknet traffic, and is independent from the nature of the source of the traffic. To the best of our knowledge, the proposed model renders a first attempt ever to formally tackle the problem of preprocessing darknet traffic. Through empirical evaluations using real darknet traffic and by comparing the proposed model against the baseline and a heuristic approach, we demonstrate the accuracy and effectiveness of the model.