Multi-level web content filter model based on MapReduce

In order to solve the problem that the traditional web filtering systems cannot filter the sensitive web pages effectively in real time, a multi-level web content filtering model based on MapReduce is proposed. Three kinds of filtering strategies are employed in this model, which are blocking of IP address and URL, keyword filtering and intelligent filtering. In intelligent filtering mechanism, the improved Knn algorithm based on maximum category space is adopted to classify the web pages intelligently and filter out the sensitive ones. To reduce the filtering time, we propose a parallelization framework based on the distributed computing MapReduce. The framework will carry out the large calculation of feature vector and Euclidean distance in parallel, which improves the filter efficiency a lot. Experimental result shows that the distributed filtering model can filter the sensitive web pages effectively in real time, and higher rate of web filtering performance with high accuracy can by achieved by the increasing of the distributed nodes.