The Discovery and Solution of the Pseudo-Reasoning Issue for Constructing Cost-Effective Multi-Agent Frameworks in Large Language Models

Large Language Models (LLMs) show exceptional logical reasoning capabilities. Despite these advancements, a significant performance and cost gap remains between smaller and larger models. Various prompting techniques have been developed to enhance LLM performance, but the performance gap remains unresolved. Additionally, multi-agent frameworks have also been introduced, however, there is often insufficient emphasis on token consumption, which impacts cost and time efficiency. This paper presents a comparative analysis of different LLMs and prompting strategies using the Counter-Intuitive AR dataset. Our findings reveal that sophisticated prompt engineering methods often result in diminishing returns and are less cost-effective than simply using larger models. This phenomenon, which we term the “pseudo-reasoning issue”, occurs when models appear to demonstrate genuine reasoning but rely on past learned patterns. We show that GPT-4o, even without additional prompting strategies, achieves 76% accuracy, surpassing all the tested methods in enhancing GPT-3.5- Turbo performance. Conversely, a heterogeneous approach, which combines large and small LLMs in solving complex problems is feasible in terms of cost-effectiveness. Our linear graph shows that increased involvement of GPT-4o in text generation correlates with higher model accuracy. These findings challenge the current trend towards increasingly complex strategies and suggest the crucial need for evaluating cost-effectiveness when designing novel prompting strategies.