Whitelists that Work: Creating Defensible Dynamic Whitelists with Statistical Learning

Blacklists are a fundamental component of many cyber security products. Typically populated from a number of sources including machine learning algorithms and malware sandboxes, large automated blacklists carry the risk of blocking access to necessary, benign Internet resources and negatively impacting users. Whitelists reduce this risk by identifying items that should not be blocked. Unfortunately, whitelists are often poorly constructed and ill-maintained, failing to provide the proper counter-balance and potentially weakening the security ecosystem. This paper introduces the first published algorithm for automated whitelist creation. Based on Bayesian statistical learning methods and incorporating network and threat information, it is both defensible and responsive to the environment. For comparison, we describe two additional algorithms for whitelisting and evaluate all three over a six-week period using multiple data sources. Further, we demonstrate how the proposed algorithm can be used as a quality assurance mechanism for blacklists, a concept not previously described in the literature.