A Human–Machine Methodology for Investigating Systems Thinking in a Complex Corpus

Systems thinking characterizes the paradigm needed to effectively design, maintain, and utilize systems. Prior work has shown that there is a language of systems thinking and that its presence can be quantified within text using supervised learning methods. Building on this foundation, we present a human-in-the-loop methodology that utilizes topic models to facilitate the identification of systems thinking within a corpus of documents. Though explorative, it requires no manual grading of documents, which makes it significantly faster than previous methods. The methodology uses each document's topic proportion within a systems thinking topic as a proxy measure for the potential of strong systems thinking. The novel aspect of the methodology is in the seeding of the corpus; the user encourages the emergence of the systems thinking topic by adding several documents that demonstrate strong systems thinking to the corpus. Additionally, seeding could be used with concepts other than systems thinking. A Tukey test on a graded corpus reveals that the top echelons of strong systems thinking papers have significantly higher mean topic proportions in the systems thinking topic than lower graded papers. Additionally, a case study on a corpus of Army documents related to the development, character, and management of soldiers demonstrates the methodology's effectiveness in overviewing a system and in providing research direction. The definition of strong systems thinking and the interpretation of topics are subjective, but the methodology overcomes this hurdle by leveraging human intuition and keeping a human in the loop.