Extraction of Archetype from Near Duplicates in Software Documentation

Software documentation contains a large amount of duplicate text, which is often comprised of near duplicates - repetitions of the same text with slight differences. They emerge due to numerous copy-pastes that have been slightly modified. Uncontrolled near duplicates complicate documentation support to a significant degree. There are some research papers on detection and management of duplicates in software documentation, but only the Duplicate Finder approach addresses the problem of near duplicates. Nevertheless, Duplicate Finder's search algorithms do not provide extraction of archetype (common text) for detected groups of near duplicates (a set of near duplicates belong to one group if they have a lot of commonalities). Archetype of group can be used in visualization of the common text and differences of duplicates for manual analysis, as well as for reuse of documentation. In this paper, we present an algorithm for archetype extraction and results of experiments on documentation of several well-known open source Java projects JUnit, Mockito, SLF4J.