Are We Modeling the Evidence or Our Own Biases? A Comparison of Conceptual Models Created from Reports

Conceptual modeling requires expertise in the application area and in modeling. Many research groups fulfill this requirement by analyzing qualitative data produced by subject matter experts and constructing their own representations of this evidence base as conceptual models. The final models are often portrayed as objective and directly based on the evidence, suggesting that modelers are merely vessels through which qualitative data becomes structured as a model. In this paper, we measure for the first time the extent to which a final model is shaped by the modeler's own interpretation. To analyze differences among modelers, we (i) compare the conceptual models produced individually in terms of structure and semantics and (ii) track knowledge provenance by automatically comparing which parts of the evidence base were utilized. Results demonstrate that modelers may interpret the evidence base differently, which stresses the need to disclose how modelers translate evidence before engaging in knowledge aggregation.