Studying the Knowledge and Behavior of ChatGPT via Journey to the West

Recent research showed that large language models (LLMs) can generate high-quality content capable of performing various NLP tasks and exhibiting remarkable performance. However, these LLMs are also known for their ridiculously wrong contents, also known as hallucinations, that significantly decrease their correctness in safety-sensitive applications. This paper aims to study the correctness of state-of-the-art (SOTA) LLMs like GPTs by investigating their behavior using simple questions. The author selects one of the most famous Chinese novels to build the question-answer pairs and study how SOTA LLMs behave against misleading information and rethinking strategies. The results indicate that both GPT-3.5 and GPT-4 are prone to be misled by simply denying their correct answers. They can also improve their correctness by rethinking and answering again without any external knowledge.