A large-scale URL filtering algorithm in high-speed flow

At present, more and more malicious websites have been emerged on the Internet. One of the main functions of Intrusion Detection System (IDS) and Intrusion Prevention System (IPS), which are usually deployed at the telecom backbone, is to detect the requests when users have been fraud to visit the malicious websites in time. According to some known URL blacklists collected by the security institutes and companies, the size of the malicious URLs has grown to tens of millions. So how to quickly and proficiently detect them has become a big challenge. In this paper, we designed a URL filtering algorithm inspired by the feature of the URL string itself. The main idea of this algorithm is to combine the hash table and AVL technique together. We have tested three aspects of our algorithm: memory occupation, pre-process time and match speed, and the results shows that the performance of our algorithm has a linear relationship with the scale of URL list set.