Board Forum Crawling: A Web Crawling Method for Web Forum

We present a new method of board forum crawling to crawl Web forum. This method exploits the organized characteristics of the Web forum sites and simulates human behavior of visiting Web forums. The method starts crawling from the homepage, and then enters each board of the site, and then crawls all the posts of the site directly. Board forum crawling can crawl most meaningful information of a Web forum site efficiently and simply. We experimentally evaluated the effectiveness of the method on real Web forum sites by comparing with the traditional breadth-first crawling. We also used this method in a real project, and 12000 Web forum sites have been crawled successfully. These results show the effectiveness of our method