Web content information extraction based on DOM tree and statistical information

Booming web pages contain a lot of information, while they contain little content and much unrelated noise information, such as script code, links, advertising and so on. These unrelated noise information occupies a lot of space, which is not suitable for the transition to small mobile devices, data mining and information retrieval. Therefore, web information extraction technology becomes more and more important. However, most extraction methods cannot adapt various and heterogeneous web structure and have poor generality and extracting efficiency. In this paper, we propose a method which can adapt to the heterogeneity and variability of web pages and gets high precision and recall. Our method is based on DOM structure to divide one web page into several blocks, and extract content blocks with statistical information instead of machine learning repeating training and manual labeling, which gets a good performance in Precision, Recall and F1.