DomainHarvester: Uncovering Trustworthy Domains Beyond Popularity Rankings

Allow lists are crucial in cybersecurity for distinguishing safe websites from potential threats. Traditional approaches relying on website popularity often fail to capture trustworthy but less-visited domains, leading to increased false positives and overlooked niche websites. This paper presents DomainHarvester, an innovative bottom-up system that leverages the web’s hyperlink structure and a Transformer-based machine learning approach to systematically identify and include these underrepresented yet legitimate domains. Results demonstrate how DomainHarvester dynamically curates an expanded allow list (DHList), substantially reducing the risk of false positives while retaining high precision in excluding malicious sites. Comprehensive evaluations and a real-world case study with a managed security services provider illustrate the efficacy and practicality of this approach. By integrating DomainHarvester, organizations and researchers can benefit from a more inclusive and globally representative cybersecurity allow list, addressing limitations in existing top-list-based solutions.