Preserving the documentary structure of source code in language-based transformation tools

Language-based tools necessarily translate textual source code into grammar-based representations. During translation, tools such as compilers and analyzers are generally free to discard comments and white space, which have no impact on the code's formal meaning. Tools that produce transformed source code for human consumption enjoy no such freedom. Comments and white space are crucial to the comprehensibility and maintainability of source code and thus to its ultimate value. However, it is not always practical or desirable for transformation tools to replicate comments and white space in their entirety. An analysis of the documentary (as opposed to linguistic) structure of source code leads to a practical strategy for preserving its comprehensibility when processed by such tools.