Logical Reasoning with LLMs via Few-Shot Prompting and Fine-Tuning: A Case Study on Turtle Soup Puzzles

Recent advancements in Large Language Models (LLMs) have greatly improved their ability to tackle complex reasoning tasks. Logical reasoning remains a crucial yet challenging endeavor. This study evaluates the performance of several leading LLMs, including OpenAI's GPT-40, Meta's Llama3.1, Mistral AI's Mistral-7B, Alibaba's Qwen2 and Qwen2.5, and Shanghai AI Laboratory's InternLM2.5, on logical reasoning challenges presented by the Turtle Soup game-a lateral thinking puzzle that demands deductive reasoning. We conducted extensive experiments using both few-shot prompting and fine-tuning, and introduced the Valid Classification Ratio (VCR) as a novel metric to assess the reliability of model outputs. Our findings reveal that while few-shot learning can be effective, fine-tuning generally yields superior results. Qwen2.5-72B achieved the highest F1 score of 80.88% through few-shot learning, surpassing GPT-40's best score of 80.36%. Llama3.1-70B also outperformed GPT-40, with an F1 score of 80.77% after fine-tuning. Additionally, InternLM2.5-7B-1M, featuring an extended context length of 1 million tokens, demonstrated significant improvement, achieving 77.01% in few-shot and 80.28% after fine-tuning, ranking third overall among fine-tuned models. These results underscore the substantial potential of latest open-source models in complex reasoning tasks, showing that they can not only match but also surpass proprietary alternatives.