A multi-layer bloom filter for duplicated URL detection

It is of great significance to improve the speed of data collecting and updating in a web crawler because there are a large number of web pages in Internet. A duplicated URL detection approach based on multi-layer bloom filter algorithm is proposed in this paper, which divides an entire URL into some layers and stores them in multi-layer bloom filter. The experimental result shows that the false positive of multi-layer bloom filter algorithm is significantly lower than that of classical bloom filter algorithm, while the efficiency of the former is almost the same as the later.