A Reference Model for Empirically Comparing LLMs with Humans

Large Language Models (LLM) have shown stunning abilities to carry out tasks that were previously conducted by humans. The future role of humans and the responsibilities assigned to non-human LLMs affect society fundamentally. In that context, LLMs have often been compared to humans. However, it is surprisingly difficult to make a fair empirical comparison between humans and LLMs. To address those difficulties, we aim at establishing a systematic approach to guide researchers in comparing LLMs with humans across various tasks. In a literature review, we examined key differences and similarities among several existing studies. We developed a reference model of the information flow based on that literature exploration. We propose a framework to support researchers in designing and executing studies, and in assessing LLMs with respect to humans. Future studies can use the reference model as guidance for designing and reporting their own unique study design. We want to support researchers and the society to take a maturation step in this emerging and constantly growing field.