Online URL Classification for Large-Scale Streaming Environments

Large-scale streaming URLs are the norm in many commercial software products that aim to filter URLs based on their sensitivity or risk level. In such problem scenarios, filtering is typically done by classifying a URL using either its webpage content or certain additional contextual information. However, such approaches are slow and computationally expensive, as they require gathering and processing webpage content or other contextual information for each URL. In this work, the authors propose a method for classifying URLs in large-scale streaming environments that doesn't suffer from these drawbacks. The proposed method is based on online ensemble learning, which results in lightweight prediction models that are well-suited for classification of streaming datasets. The authors illustrate the effectiveness of the proposed approach using large-scale datasets from a live, production environment and show that the proposed method results in an increase of 5 to 8 percent in terms of precision and 3 to 5.5 percent in terms of recall.