DOM-Based XHTML Document Structure Analysis Separating Content from Navigation Elements

This paper describes an algorithm that attempts to distinguish core content from clutter within a web document. The end goal is to aid in the separation of the core-content from hyperlinked-clutter such as text advertisements and long links of syndicated references to other web documents. Its advantage over other approaches is its ability to identify both loosely as well as tightly defined "table-like" or "list-like" structures of hyperlinks (from nested tables to simple, bullet-pointed lists) by operating at various levels within the DOM tree. The resulting data can then be used to "narrow down" the core-content of a web document for semantic analysis or other information retrieval purposes as well as to aid in the process of "clipping" a web document to its bare essentials for use with hardware-limited devices such as PDAs and cell phones.