Research on improved focused crawler and its application in food safety public opinion analysis

In big data environment, the performance of the focused crawler has a great impact on the results of crawling. In order to improve the efficiency and accuracy of focused crawler, this paper proposed a combination method of HTML analysis and text density and multi-reference factors similarity calculation method based on the basic principle and key technologies of focused crawler. The proposed method calculated the similarity of web page by the page content and crawler theme. This method can significantly improve the accuracy of web text density of focused crawler. Meanwhile, the proposed method also optimized the initial seed module and dynamic threshold module to implement a new web focused crawler system for specific topics. Finally, this paper also used the improved crawler, Best-First crawler and Breadth-First crawler to collect the public opinion data of the big data with the topic of food safety and analyzed the performance and efficiency of the crawling results. The results show that the improved crawler can improve the feasibility and efficiency compared with the other two crawlers. Moreover, this paper proved that the focused crawler based on text density and multi-factor similarity calculate can be applied efficiently in food safety public opinion analysis.