Automatic Online Monitoring and Data-Mining Internet Forums

With the advancement of internet technology and the change in the mode of communication, it is found that much first-hand news have been discussed in Internet forums well before they are reported in traditional mass media. Also, this communication channel provides an effective channel for illegal activities such as dissemination of copyrighted movies, threatening messages and online gambling etc. The law enforcement agencies are looking for solutions to monitor these discussion forums for possible criminal activities and download suspected postings as evidence for investigation. The volume of postings is huge, for 10 popular forums in Hong Kong, we found that there are 300,000 new messages every day. In this paper, we propose an automatic system that tackles this problem. Our proposed system will download postings from selected discussion forums continuously and employ data mining techniques to identify hot topics and cluster authors into different groups using word-based user profiles. Difference techniques are applied to process the collected data and several ways are proposed to solve the problem.