Web informative content block detecting based on entropy and parent-child relationship in DOM

To increase the commercial value and accessibility of pages, most sites tend to publish their pages with redundant information, such as navigation panels, advertisements, and copyright announcements. Such redundant information almost exists in all pages of the website, which increases the index size of general search engines and causes page topics to drift. In this paper, We propose an informative content blocks detecting system called WICBDPCR(Web Informative Content Block Detecting based on Parent-Child Relationship in the document object model) which applies Information Theory to DOM tree in order to detect the informative structure. Experiments on several real commercial Web sites show high precision and recall rates, which validate WICBDPCR’s practical applicability.