A comparative study with human data: Do LLMs have superficial language processing?

This study aims to test the superficial language processing hypothesis on a natural language generation (NLG) task with large language models (LLMs) and compare it with human performance. We examined whether LLMs sometimes process sentences superficially like humans or always have deepsyntactic processing. The good-enough language processing experiment conducted by Ferreira et al. with English speakers was repeated in this study with OpenAI"s GPT-3.5-turbo and textdavinci- 003 models. The grammatical accuracy of the answers given by the LLMs was compared with the average Surprisal (negative log-likelihood) of the English experimental sentences we predicted with RoBERTa, and with human data. We found that the GPT-3.5-turbo model was more accurate than the textdavinci-003 model, but was less successful than humans in processing sentences with semantic anomalies and these models were not able to do abstraction like humans. When compared with Surprisal values, we found that the GPT-3.5-turbo model processed sentences by paying more attention to syntactic cues, while the text-davinci- 003 model processed sentences superficially.