Performance of Large Language Models and Analysis of Responses in the Wisconsin Card Sorting Task

Cognitive flexibility, the ability to adapt rules and tasks in response to changing environments, is an important metric for evaluating large language models (LLMs). Previous research suggests that LLMs may struggle with error correction and the flexible application of rules; however, their underlying mechanisms remain unclear, partly because cognitive flexibility consists of multiple components. In this study, following the framework of “machine psychology,” which employs psychological experiments designed for humans to investigate the information-processing mechanisms of LLMs, we applied the Wisconsin card sorting task (WCST) to various LLMs. Our results showed that ChatGPT o1 achieved correct response rates above $50 \%$, whereas ChatGPT o1, ChatGPT-o1 mini, ChatGPT 40-mini, Gemini 1.5, and Gemini 2.0 recorded rates below 30%. Notably, LLMs exhibited a rule-response mismatch when identifying the task-solving rules, with the tendency of the mismatch differing between models. These findings suggest that the tendency of the rule-estimation and the integration of bottomup card information and top-down rule-based reasoning differs between models, contributing to discussions on the mechanisms underlying the lack of cognitive flexibility in LLMs and humans and potential ways to improve their performance.